Top

World Wide Web

Published in:

01-03-2024

PriMonitor: An adaptive tuning privacy-preserving approach for multimodal emotion detection

Authors: Lihua Yin, Sixin Lin, Zhe Sun, Simin Wang, Ran Li, Yuanyuan He

Published in: World Wide Web | Issue 2/2024

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

The proliferation of edge computing and the Internet of Vehicles (IoV) has significantly bolstered the popularity of deep learning-based driver assistance applications. This has paved the way for the integration of multimodal emotion detection systems, which effectively enhance driving safety and are increasingly prevalent in our daily lives. However, the utilization of in-vehicle cameras and microphones has raised concerns regarding the extensive collection of driver privacy data. Applying privacy-preserving techniques to a single modality alone proves insufficient in preventing privacy re-identification when correlated with other modalities. In this paper, we introduce PriMonitor, an adaptive tuning privacy-preserving approach for multimodal emotion detection. PriMonitor tackles these challenges by proposing a generalized random response-based differential privacy method that not only enhances the speed and data availability of text privacy protection but also ensures privacy preservation across multiple modalities. To determine suitable weight assignments within a given privacy budget, we introduce pre-aggregator and iterative mechanisms. Our PriMonitor effectively mitigates privacy re-identification due to modal correlation while maintaining a high level of accuracy in multimodal models. Experimental results validate the efficiency and competitiveness of our approach.

previous article Multi-stage dynamic disinformation detection with graph entropy guidance

next article Invariant representation learning to popularity distribution shift for recommendation

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Oham, C., Michelin, R.A., Jurdak, R., Kanhere, S.S., Jha, S.: B-FERL: Blockchain based framework for securing smart vehicles. Inf. Process. Manage. 58(1), 102426 (2021)CrossRef

Li, L., Liu, J., Cheng, L., Qiu, S., Wang, W., Zhang, X., Zhang, Z.: Creditcoin: A privacy-preserving blockchain-based incentive announcement network for communications of smart vehicles. In: IEEE Transactions on Intelligent Transportation Systems, 19(7), 2204–2220 (2018)

Sun, Y., Tian, Z., Li, M., Zhu, C., Guizani, N.: Automated attack and defense framework toward 5G security. IEEE Netw. 34(5), 247–253 (2020)CrossRef

Sicari, S., Rizzardi, A., Coen-Porisini, A.: 5G in the internet of things era: an overview on security and privacy challenges. Comput. Netw. 179, 107345 (2020)CrossRef

Qiao, C., Brown, K.N., Zhang, F., Tian, Z.: Adaptive Asynchronous Clustering Algorithms for Wireless Mesh Networks. In: IEEE Transactions on Knowledge and Data Engineering, 35(3), 2610–2627 (2023)

Huang, J., Li, Y., Tao, J., Lian, Z., Niu, M., Yang, M.: Multimodal continuous emotion recognition with data augmentation using recurrent neural networks. In: Proceedings of the 2018 on Audio/Visual Emotion Challenge and Workshop, pp. 57–64 (2018)

Aguilar, G., Rozgic, V., Wang, W., Wang, C.: Multimodal and Multi-view Models for Emotion Recognition. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 991–1002 (2019)

Ma, M., Ren, J., Zhao, L., Tulyakov, S., Wu, C., Peng, X.: SMIL: Multimodal learning with severely missing modality. In: Proceedings of the AAAI Conference on Artificial Intelligence. 35(3), 2302–2310 (2021)

Li, A., Duan, Y., Yang, H., Chen, Y., Yang, J.: TIPRDC: task-independent privacy-respecting data crowdsourcing framework for deep learning with anonymized intermediate representations. In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 824–832 (2020)

10.

Yin, L., Feng, J., Xun, H., Sun, Z., Cheng, X.: A privacy-preserving federated learning for multiparty data sharing in social IoTs. IEEE Trans. Netw. Sci. Eng. 8(3), 2706–2718 (2021)CrossRef

11.

Fernandes, N., Dras, M., McIver, A.: Generalised differential privacy for text document processing. In: International Conference on Principles of Security and Trust, pp. 123–148 (2019)

12.

He, Y., Tan, X., Ni, J., Yang, L.T., Deng, X.: Differentially Private Set Intersection for Asymmetrical ID Alignment. In: IEEE Transactions on Information Forensics and Security. 17, 3479–3494 (2022)

13.

Wang, T., Blocki, J., Li, N., Jha, S.: Locally differentially private protocols for frequency estimation. In: 26th USENIX Security Symposium, pp. 729–745 (2017)

14.

Zepf, S., Hernandez, J., Schmitt, A., et al.: Driver emotion recognition for intelligent vehicles: A survey. ACM Comput. Surv. (CSUR) 53(3), 1–30 (2020)CrossRef

15.

Ma, Z., Mahmoud, M., Robinson, P., Dias, E., Skrypchuk, L.: Automatic detection of a driver’s complex mental states. In: Computational Science and Its Applications - ICCSA 2017: 17th International Conference, Trieste, Italy, July 3–6, 2017, Proceedings, Part III. Series: Lecture notes in computer science (10406). Springer: Cham, pp. 678–691 (2017)

16.

Karimi, S., Sedaaghi, M.H.: Robust emotional speech classification in the presence of babble noise. Int. J. Speech Technol. 16, 215–227 (2013)CrossRef

17.

Deng, Y., Wu, Z., Chu, C-H., Zhang, Q., Hsu, D. F.: Sensor feature selection and combination for stress identification using combinatorial fusion. Int. J. Adv. Robot. Syst. 306–313 (2013)

18.

Karaduman, O., Eren, H., Kurum, H., Celenk, M.: An effective variable selection algorithm for aggressive/calm driving detection via CAN bus. In: Proceedings of the 2013 International Conference on Connected Vehicles and Expo (ICCVE’13), Las Vegas, NV, USA, 2013, pp. 586–591 (2013)

19.

Hazarika, D., Poria, S., Mihalcea, R., Cambria, E., Zimmermann, R.: Icon: Interactive conversational memory network for multimodal emotion detection. In: Proceedings of the 2018 conference on empirical methods in natural language processing, pp. 2594–2604 (2018)

20.

Mittal, T., Guhan, P., Bhattacharya, U., Chandra, R., Bera, A., Manocha, D.: Emoticon: Context-aware multimodal emotion recognition using frege's principle. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14234–14243 (2020)

21.

Yu, W., Xu, H., Meng, F., Zhu, Y., Ma, Y., Wu, J., Zou, J. Yang, K.: CH-SIMS: A Chinese multimodal sentiment analysis dataset with fine-grained annotation of modality. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 3718–3727 (2020)

22.

Zheng, W., Yu, J., Xia, R., Wang, S.: A Facial Expression-Aware Multimodal Multi-task Learning Framework for Emotion Recognition in Multi-party Conversations. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, pp. 15445–15459 (2023)

23.

Illendula, A., Sheth, A.: Multimodal emotion classification. In: companion proceedings of the 2019 world wide web conference, pp. 439–449 (2019)

24.

Ma, J., Tang, H., Zheng, W. L., Lu, B. L.: Emotion recognition using multimodal residual LSTM network. In: Proceedings of the 27th ACM international conference on multimedia, pp. 176–183 (2019)

25.

Lv, F., Chen, X., Huang, Y., Duan, L., Lin, G.: Progressive modality reinforcement for human multimodal emotion recognition from unaligned multimodal sequences. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2554–2562 (2021)

26.

Mittal, T., Bhattacharya, U., Chandra, R., Bera, A., Manocha, D.: M3er: Multiplicative multimodal emotion recognition using facial, textual, and speech cues. In: Proceedings of the AAAI conference on artificial intelligence, Vol. 34, No. 02, pp. 1359–1367 (2020)

27.

Firdaus, M., Chauhan, H., Ekbal, A., Bhattacharyya, P.: MEISD: a multimodal multi-label emotion, intensity and sentiment dialogue dataset for emotion recognition and sentiment analysis in conversations. In: Proceedings of the 28th International Conference on Computational Linguistics, pp. 4441–4453 (2020)

28.

Liang, B., Lou, C., Li, X., Yang, M., Gui, L., He, Y., Pei, W., Xu, R.: Multi-Modal Sarcasm Detection via Cross-Modal Graph Convolutional Network. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pp. 1767–1777 (2022)

29.

Yang, H., Gao, X., Wu, J., Gan, T., Ding, N., Jiang, F., Nie, L.: Self-adaptive Context and Modal-interaction Modeling For Multimodal Emotion Recognition. In: Findings of the Association for Computational Linguistics: ACL 2023, pp. 6267–6281 (2023)

30.

Tan, Q., Wang, X., Shi, W., Tang, J., Tian, Z.: An Anonymity Vulnerability in Tor. In: IEEE/ACM Transactions on Networking. 30(6), 2574–2587 (2022)

31.

Jaiswal, M., Provost, E. M.: Privacy enhanced multimodal neural representations for emotion recognition. In: Proceedings of the AAAI Conference on Artificial Intelligence. 34(05), 7985–7993 (2020)

32.

Gupta, A., Tafasca, S., Odobez, J. M.: A Modular Multimodal Architecture for Gaze Target Prediction: Application to Privacy-Sensitive Settings. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 5041–5050 (2022)

33.

Zhao, L., Ni, L., Hu, S., Chen, Y., Zhou, P., Xiao, F., Wu, L.: Inprivate digging: Enabling tree-based distributed data mining with differential privacy. In: IEEE INFOCOM 2018-IEEE Conference on Computer Communications, pp. 2087–2095 (2018)

34.

Papernot, N., Thakurta, A., Song, S., Chien, S., Erlingsson, Ú.: Tempered sigmoid activations for deep learning with differential privacy. In: Proceedings of the AAAI Conference on Artificial Intelligence. 35(10), 9312–9321 (2021)

35.

Sun, Y., Tian, Z., Li, M., Su, S., Du, X., Guizani, M.: Honeypot identification in softwarized industrial cyber–physical systems. IEEE Trans. Industr. Inf. 17(8), 5542–5551 (2020)CrossRef

36.

Chai, Y., Du, L., Qiu, J., Yin, L., Tian, Z.: Dynamic prototype network based on sample adaptation for few-shot malware detection. In: IEEE Trans. Know. Data Eng. 4754–4766 (2020)

37.

Arcolezi, H. H., Gambs, S., Couchot, J. F., Palamidessi, C.: On the Risks of Collecting Multidimensional Data Under Local Differential Privacy. In: Proceedings of the VLDB Endowment (PVLDB). 16 (5), 1126–1139 (2023)

38.

Croft, W.L., Sack, J.R., Shi, W.: Obfuscation of images via differential privacy: From facial images to general images. Peer Peer Netw. Appl. 14(3), 1705–1733 (2021)CrossRef

39.

Heikkilä, M., Lagerspetz, E., Kaski, S., Shimizu, K., Tarkoma, S., Honkela, A.: Differentially private bayesian learning on distributed data. In: Proceedings of the 31st International Conference on Neural Information Processing Systems December 2017 pp. 3229–3238 (2017)

40.

Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., Zhang, L.: Deep learning with differential privacy. In: Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp. 308–318 (2016)

41.

Xu, C., Ren, J., Zhang, D., Zhang, Y., Qin, Z., Ren, K.: GANobfuscator: Mitigating information leakage under GAN via differential privacy. IEEE Trans. Inf. Forensics Secur. 14(9), 2358–2371 (2019)CrossRef

42.

Hao, M., Li, H., Xu, G., Liu, S., Yang, H.: Towards efficient and privacy-preserving federated deep learning. In: ICC 2019–2019 IEEE international conference on communications (ICC), pp. 1–6 (2019)

43.

Yang, R., Ma, X., Bai, X., Su, X.: Differential privacy images protection based on generative adversarial network. In: 2020 IEEE 19th International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom), pp. 1688–1695 (2020)

44.

Feyisetan, O., Balle, B., Drake, T., Diethe, T.: Privacy-and utility-preserving textual analysis via calibrated multivariate perturbations. In: Proceedings of the 13th International Conference on Web Search and Data Mining, pp. 178–186 (2020)

45.

Lyu, L., Li, Y., He, X., Xiao, T.: Towards differentially private text representations. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1813–1816 (2020)

46.

Feyisetan, O., Diethe, T., Drake, T.: Leveraging hierarchical representations for preserving privacy and utility in text. In: 2019 IEEE International Conference on Data Mining (ICDM), pp. 210–219 (2019)

47.

Weggenmann, B., Kerschbaum, F.: Syntf: Synthetic and differentially private term frequency vectors for privacy-preserving text mining. In: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 305–314 (2018)

48.

Devlin, J., Chang, M. W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4171– 4186 (2019)

49.

Yue, X., Du, M., Wang, T., Li, Y., Sun, H., Chow, S.S.: Differential Privacy for Text Analytics via Natural Text Sanitization. In: Findings of the Association for Computational Linguistics: ACL-IJCNLP pp. 3853–3866 (2021)

50.

Cormode, G., Jha, S., Kulkarni, T., Li, N., Srivastava, D., Wang, T.: Privacy at scale: Local differential privacy in practice. In: Proceedings of the 2018 International Conference on Management of Data, pp. 1655–1658 (2018)

51.

Liu, Y., Peng, J., Kang, J., Iliyasu, A.M., Niyato, D., Abd El-Latif, A.: A secure federated learning framework for 5G networks. IEEE Wirel. Commun. 27(4), 24–31 (2020)CrossRef

52.

Li, B., Hou, Y., Che, W.: Data augmentation approaches in natural language processing: A survey. AI Open 3, 71–90 (2022)CrossRef

53.

Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 25st International Conference on Neural Information Processing Systems December 2011 pp. 2546–2554 (2011)

54.

Zadeh, A., Chen, M., Poria, S., Cambria, E., Morency, L. P.: Tensor fusion network for multimodal sentiment analysis. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 1103–1114 (2017)

55.

Liu, Z., Shen, Y., Lakshminarasimhan, V. B., Liang, P. P., Zadeh, A., Morency, L. P.: Efficient low-rank multimodal fusion with modality-specific factors. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2247–2256 (2018)

56.

Zadeh, A., Liang, P. P., Mazumder, N., Poria, S., Cambria, E., Morency, L. P.: Memory fusion network for multi-view sequential learning. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 1103–1114 (2018)

57.

Zadeh, A.B., Liang, P.P., Poria, S., Cambria, E., Morency, L.P.: Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2236–2246 (2018)

58.

Ye, J., Maddi, A., Murakonda, S.K., Bindschaedler, V., Shokri, R.: Enhanced Membership Inference Attacks against Machine Learning Models. In: Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security (CCS '22), pp. 3093–3106 (2022)

Title: PriMonitor: An adaptive tuning privacy-preserving approach for multimodal emotion detection
Authors: Lihua Yin
Sixin Lin
Zhe Sun
Simin Wang
Ran Li
Yuanyuan He
Publication date: 01-03-2024
Publisher: Springer US
Published in: World Wide Web / Issue 2/2024
Print ISSN: 1386-145X
Electronic ISSN: 1573-1413
DOI: https://doi.org/10.1007/s11280-024-01246-7

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft"

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Other articles of this Issue 2/2024

Efficiently estimating node influence through group sampling over large graphs

Multi-stage dynamic disinformation detection with graph entropy guidance

LAMEE: a light all-MLP framework for time series prediction empowering recommendations

Modeling dynamic spatiotemporal user preference for location prediction: a mutually enhanced method

High-level preferences as positive examples in contrastive learning for multi-interest sequential recommendation

Path-based approximate matching of fuzzy spatiotemporal RDF data

Premium Partner