Anna-Marie Ortloff

I have been a Phd student in the Behavioral Security and Privacy group at the University of Bonn, under the supervision of Matthew Smith, since January 2021. My research focuses on methodological issues around both qualitative and quantative research methods used in HCI, and more specifically in the domain of Usable Security and Privacy. I apply the research methods I investigate in various domains at the intersection of usability, privacy, security, and human information behavior.

In 2020, I finished my Master's degree of Media Informatics at the University of Regensburg, Germany with my thesis about "A Comparison of Nudging and Boosting for Privacy during Web Browsing". Prior to that, in 2018, I completed a Bachelor's degree in Information Science (major), Media Informatics (minor) and Spanish (minor), also at the University of Regensburg.

last at cs.uni-bonn.de

Publications

Anna-Marie Ortloff, Christian Tiefenau & Matthew Smith
SoK: I Have the (Developer) Power! Sample Size Estimation for Fisher's Exact, Chi-Squared, McNemar's, Wilcoxon Rank-Sum, Wilcoxon Signed-Rank and t-tests in Developer-Centered Usable Security
(2023) Symposium on Usable Privacy and Security (SOUPS'23)
Distinguished Paper Award

[Bibtex]
[PDF]
[Abstract]
[Presentation Video] [Companion website]

@inproceedings{Ortloff2023_DeveloperPower,
author = {Anna-Marie Ortloff and Christian Tiefenau and Matthew Smith},
title = {{SoK}: I Have the (Developer) Power! Sample Size Estimation for Fisher{\textquoteright}s Exact, {Chi-Squared}, {McNemar{\textquoteright}s}, Wilcoxon {Rank-Sum}, Wilcoxon {Signed-Rank} and t-tests in {Developer-Centered} Usable Security},
booktitle = {Nineteenth Symposium on Usable Privacy and Security (SOUPS 2023)},
year = {2023},
isbn = {978-1-939133-36-6},
address = {Anaheim, CA},
pages = {341--359},
url = {https://www.usenix.org/conference/soups2023/presentation/ortloff},
publisher = {USENIX Association},
}

A priori power analysis would be very beneficial for researchers in the field of developer-centered usable security since recruiting developers for studies is challenging. Power analysis allows researchers to know how many participants they need to test their null hypotheses. However, most studies in this field do not report having conducted power analysis. We conducted a meta-analysis of 54 top-tier developer study papers and found that many are indeed underpowered even to detect large effects. To aid researchers in conducting a priori power analysis in this challenging field, we conducted a systematization of knowledge to extract and condense the needed information. We extracted information from 467 tests and 413 variables and developed a data structure to systematically represent information about hypothesis tests, involved variables, and study methodology. We then systematized the information for tests with categorical independent variables with two groups, i.e., Fisher's exact, chi-squared, McNemar's, Wilcoxon rank-sum, Wilcoxon signed-rank, and paired and independent t-tests to aid researchers with power analysis for these tests. Additionally, we present overview information on the field of developer-centered usable security and list recommendations for suitable reporting practices to make statistical information for power analysis and interpretation more accessible for researchers.
Anna-Marie Ortloff, Matthias Fassl, Alexander Ponticello, Florin Martius, Anne Mertens, Katharina Krombholz & Matthew Smith
Different Researchers, Different Results? Analyzing the Influence of Researcher Experience and Data Type During Qualitative Analysis of an Interview and Survey Study on Security Advice
(2023) Conference on Human Factors in Computing Systems (CHI'23)

[Bibtex]
[PDF]
[Abstract]
[supplemental]

@inproceedings{Ortloff2023_DifferentResearchers,
author = {Ortloff, Anna-Marie and Fassl, Matthias and Ponticello, Alexander and Martius, Florin and Mertens, Anne and Krombholz, Katharina and Smith, Matthew},
title = {Different Researchers, Different Results? Analyzing the Influence of Researcher Experience and Data Type During Qualitative Analysis of an Interview and Survey Study on Security Advice},
year = {2023},
isbn = {9781450394215},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3544548.3580766},
doi = {10.1145/3544548.3580766},
abstract = {When conducting qualitative research it is necessary to decide how many researchers should be involved in coding the data: Is one enough or are more coders beneficial? To offer empirical evidence for this question, we designed a series of studies investigating qualitative coding. We replicated and extended a usable security and privacy study by Ion et al. to gather both simple survey data and complex interview data. We had a total of 65 students and seven researchers analyze different parts of this data. We analyzed the codebook creation process, similarity of outcomes, inter-rater reliability, and compared the student to the researcher outcomes. We also surveyed five years of SOUPS-PC members about their views on coding. The reviewers view on coding practices for complex and simple data are almost identical. However, our results suggest that the coding process can be different for the two types of data, with complex data benefiting more from interaction between coders.},
booktitle = {Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems},
articleno = {864},
numpages = {21},
keywords = {quality criteria, qualitative analysis, reliability},
location = {Hamburg, Germany},
series = {CHI '23}
}

When conducting qualitative research it is necessary to decide how many researchers should be involved in coding the data: Is one enough or are more coders beneficial? To offer empirical evidence for this question, we designed a series of studies investigating qualitative coding. We replicated and extended a usable security and privacy study by Ion et al. to gather both simple survey data and complex interview data. We had a total of 65 students and seven researchers analyze different parts of this data. We analyzed the codebook creation process, similarity of outcomes, inter-rater reliability, and compared the student to the researcher outcomes. We also surveyed five years of SOUPS-PC members about their views on coding. The reviewers view on coding practices for complex and simple data are almost identical. However, our results suggest that the coding process can be different for the two types of data, with complex data benefiting more from interaction between coders.
Lisa Geierhaas, Anna-Marie Ortloff, Matthew Smith & Alena Naiakshina
Let's Hash: Helping Developers with Password Security
(2022) Symposium on Usable Privacy and Security (SOUPS'22)
Distinguished Paper Award

[Bibtex]
[PDF]
[Abstract]
[Presentation Video]

@inproceedings{Geierhaas2022_LetsHash,
title = {Let{\textquoteright}s} Hash: Helping Developers with Password Security},
author = {Geierhaas, Lisa and Ortloff, Anna-Marie and Smith, Matthew and Naiakshina, Alena},
year = {2022},
isbn = {978-1-939133-30-4},
publisher = {USENIX Association},
url = {},
doi = {},
booktitle = {Eighteenth Symposium on Usable Privacy and Security (SOUPS 2022)},
pages = {503--522},
location = {Boston, USA},
series = {SOUPS '22}
}

Software developers are rarely security experts and often struggle with security-related programming tasks. The resources developers use to work on them, such as Stack-Overflow or Documentation, have a significant impact on the security of the code they produce. However, work by Acar et al. [4] has shown that these resources are often either easy to use but insecure or secure but hard to use. In a study by Naiakshina et al. [44], it was shown that developers who did not use resources to copy and paste code did not produce any secure solutions at all. This highlights how essential programming resources are for security. Inspired by the Let’s Encrypt and Certbot that support admins in configuring TLS, we created a programming aid called Let’s Hash to help developers create secure password authentication code easily. We created two versions. The first is a collection of code snippets developers can use, and the second adds a wizard interface on top that guides developers through the decisions which need to be made and creates the complete code for them. To evaluate the security and usability of Let’s Hash, we conducted a study with 179 freelance developers, asking them to solve three password programming tasks. Both versions of Let’s Hash significantly outperformed the baseline condition in which developers used their regular resources. On average, Let’s Hash users were between 5 and 32 times as likely to create secure code than those in the control condition.
Maximiliane Windl, Anna-Marie Ortloff, Valentin Schwind & Niels Henze
Privacy at a Glance: A Process to Learn Modular Privacy Icons During Web Browsing
(2022) ACM SIGIR Conference on Human Information Interaction and Retrieval. (CHIIR'22)

[Bibtex]
[PDF]
[Abstract]
[Presentation Video]

@inproceedings{Windl2022_PrivacyGlance,
title = {Privacy at a Glance: A Process to Learn Modular Privacy Icons During Web Browsing},
author = {Windl, Maximiliane and Ortloff, Anna-Marie and Henze, Niels and Schwind, Valentin},
year = {2022},
isbn = {9781450391863},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3498366.3505813},
doi = {10.1145/3498366.3505813},
booktitle = {ACM SIGIR Conference on Human Information Interaction and Retrieval},
pages = {102–112},
numpages = {11},
keywords = {privacy policy, privacy, icons, online study, web browsing, survey},
location = {Regensburg, Germany},
series = {CHIIR '22}
}

Privacy policies (PPs) are currently the only way to inform users about their rights and choices during web browsing and searching. However, users avoid engaging with them, because of their length and abstract legal language, which makes them hard to read and understand. We propose to support the understanding of PPs by using modular icons. Icons have already proven to be helpful in visualizing concepts with high information density. However, the value of using icons to supplement PPs lacks a scientific foundation. Thus, we conducted two studies to evaluate existing icon sets for their understandability and to teach participants their meaning in situ. We show that modular privacy icons can be taught using our process, which has the potential to aid quicker and easier comprehension of PPs. We contribute a set of tested modular privacy icons and a verified process on how to teach them to users incidentally during web browsing.
Anna-Marie Ortloff, Maike Vossen & Christian Tiefenau
Replicating a Study of Ransomware in Germany
(2021) European Symposium on Usable Security (EuroUSEC'21)

[Bibtex]
[PDF]
[Abstract]

@inproceedings{Ortloff2021_ReplicatingStudy,
title = {Replicating a Study of Ransomware in Germany},
author = {Ortloff, Anna-Marie and Vossen, Maike and Tiefenau, Christian},
year = {2021},
isbn = {978-1-4503-8423-0/21/10},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3481357.3481508},
doi = {10.1145/3481357.3481508},
booktitle = {European Symposium on Usable Security 2021},
numpages = {23},
location = {Karlsruhe, Germany},
series = {EuroUSEC '21}
}

Ransomware is a pertinent threat to businesses and end user data. While several attacks on enterprises are reported and give insights into the prevalence of such ransomware attacks, this prevalence in the general population is hard to estimate since they are not monitored by an official entity. A 2019 study by Simoiu et al. surveyed a representative sample of American consumers to estimate it for the US population. One year later, we aimed to replicate this effort for a representative German population (N=963) to study the spread of ransomware in a different context. Our findings suggest some differences between the two samples concerning payment methods and the participants' way of dealing with ransomware. Other aspects, like the ransom amounts and behavioral changes after an attack, were largely similar. We extend prior work by examining disagreements and uncertainty in judging whether a ransomware attack occurred for participants and researchers alike.
Anna-Marie Ortloff, Steven Zimmerman, David Elsweiler & Niels Henze
The Effect of Nudges and Boosts on Browsing Privacy in a Naturalistic Environment
(2021) Proceedings of the Conference on Human Information Interaction and Retrieval. (CHIIR'21)

[Bibtex]
[PDF]
[Abstract]
[Presentation Video]

@inproceedings{Ortloff2021_NudgeBoost,
title = {The Effect of Nudges and Boosts on Browsing Privacy in a Naturalistic Environment},
author = {Ortloff, Anna-Marie and Zimmerman, Steven and Elsweiler, David and Henze, Niels},
year = {2021},
isbn = {9781450380553},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3406522.3446014},
doi = {10.1145/3406522.3446014},
booktitle = {Proceedings of the 2021 Conference on Human Information Interaction and Retrieval},
pages = {63–73},
numpages = {11},
keywords = {privacy, web browsing, human information behavior, boosts, nudges},
location = {Canberra ACT, Australia},
series = {CHIIR '21}
}

During everyday web browsing and search users reveal many pieces of private information to third parties. Even though people report being concerned about their privacy online, they often do not take steps to protect it. This is known as the 'privacy paradox' in the literature. In this work we study two well-known strategies based on theories from the behavioral sciences, nudging and boosting, which encourage users to browse in a way that their private data are less exposed. First, an online survey (N=127) tested the comprehensibility and efficacy of various facts (boosts), before the most effective of these were evaluated against 'nudge' interventions previously shown to be efficacious in lab-studies. A three week naturalistic study (N=68) using a browser extension revealed that both nudges and boosts improve browsing privacy, as approximated by different measures. Boosts are also shown to improve user knowledge about privacy in the short term, but the benefit weakens over time.
Anna-Marie Ortloff, Maximiliane Windl, Valentin Schwind & Niels Henze
Implementation and In Situ Assessment of Contextual Privacy Policies
(2020) Proceedings of the Designing Interactive Systems Conference. (DIS'20)

[Bibtex]
[PDF]
[Abstract]

@inproceedings{Ortloff2020_CPP,
title = {Implementation and In Situ Assessment of Contextual Privacy Policies},
author = {Anna-Marie Ortloff and Maximiliane Windl and Valentin Schwind and Niels Henze},
doi = {10.1145/3357236.3395549},
isbn = {9781450369749},
year = {2020},
date = {2020-07-06},
booktitle = {Proceedings of the 2020 Designing Interactive Systems Conference},
numpages = {14},
publisher = {Association for Computing Machinery},
address = {Eindhoven, Netherlands},
series = {DIS ’20}
}

Online services collect an increasing amount of data about their users. Privacy policies are currently the only common way to inform users about the kinds of data collected, stored and processed by online services. Previous work showed that users do not read and understand privacy policies, due to their length, difficult language, and often non-prominent location. Embedding privacy-relevant information directly in the context of use could help users understand the privacy implications of using online services. We implemented Contextual Privacy Policies (CPPs) as a browser extension and provide it to the community to make privacy information accessible for end-users. We evaluated CPPs through a one-week deployment and in situ questionnaires as well as pre- and post-study interviews. We found that CPPs were well received by participants. The analysis revealed that provided information should be as compact as possible, be adjusted to user groups and enable users to take action.
Anna-Marie Ortloff, Maximiliane Windl, Lydia Güntner & Thomas Schmidt
Towards a Graphical User Interface for Quantitative Analysis in Digital Musicology
(2019) Mensch und Computer, workshop paper (MuC '19)

[Bibtex]
[PDF]
[Abstract]

@inproceedings{Ortloff2019_BeyondTheNotes,
author = {Ortloff, Anna-Marie and Windl, Maximiliane and Güntner, Lydia and Schmidt, Thomas},
title = {Towards a Graphical User Interface for Quantitative Analysis in Digital Musicology},
booktitle = {Mensch und Computer 2019 - Workshopband},
year = {2019},
editor = {Alt, Florian and Bulling, Andreas and Döring, Tanja},
doi = {10.18420/muc2019-ws-568},
publisher = {Gesellschaft für Informatik e.V.},
address = {Bonn}
}

We introduce the first prototype of a web application for digital musicology: BeyondTheNotes (working title). The goal of the tool is to support the in-depth analysis of individual pieces of music as well as the large scale analysis and comparison of summative features of multiple pieces of music. In contrast to existing tools, BeyondTheNotes is ready to use without installation, enables the upload and analysis of own material and offers different visualizations of musical metrics like chords, pitches, durations and key. We design the tool according to the User Centered Design Approach to improve the usability and address the specific needs of musicologists. We describe the results of the requirement analysis and discuss future steps.
Anna-Marie Ortloff, Lydia Güntner, Maximiliane Windl, Thomas Schmidt, Martin Kocur & Christian Wolff
SentiBooks: Enhancing audiobooks via affective computing and smart light bulbs
(2019) Proceedings of Mensch und Computer (MuC '19)

[Bibtex]
[PDF]
[Abstract]

@inproceedings{Ortloff2019_Sentibooks,
author = {Ortloff, Anna-Marie and Güntner, Lydia and Windl, Maximiliane and Schmidt, Thomas and Kocur, Martin and Wolff, Christian},
title = {SentiBooks: Enhancing Audiobooks via Affective Computing and Smart Light Bulbs},
booktitle = {Mensch und Computer 2019 - Tagungsband},
year = {2019},
editor = {Alt, Florian and Bulling, Andreas and Döring, Tanja},
doi = { 10.1145/3340764.3345368 },
publisher = {ACM},
address = {New York}
}

We present SentiBooks, a smartphone application to enhance the audiobook listening experience via affective computing and smart light bulbs. Users can connect to Philips Hue Light Bulbs with a smartphone app while listening to an audiobook. The app analyzes the emotional expression of the narrator of the audiobook using speech emotion recognition and adjusts the colors of the lighting settings according to the expression of the narrator in 10-seconds intervals. By transitioning between colors that are connected to the specific emotion that is currently dominant in the reading, the overall audiobook experience is intensified.
Anna-Marie Ortloff, Lydia Güntner, Maximiliane Windl, Denis Feth & Svenja Polst
Evaluation kontextueller Datenschutzerklärungen
(2018) Mensch und Computer, workshop paper (MuC '18)

[Bibtex]
[PDF]
[Abstract]

@inproceedings{Ortloff2018_CPP,
author = {Ortloff, Anna-Marie and Güntner, Lydia and Windl, Maximiliane and Feth, Denis and Polst, Svenja},
title = {Evaluation kontextueller Datenschutzerklärungen},
booktitle = {Mensch und Computer 2018 - Workshopband},
year = {2018},
editor = {Dachselt, Raimund and Weber, Gerhard},
doi = {10.18420/muc2018-ws08-0541},
publisher = {Gesellschaft für Informatik e.V.},
address = {Bonn}
}

Datenschutzerklärungen sind häufig schwer zu finden und zu verstehen. Daher lesen viele Nutzer sie nur teilweise oder gar nicht. Kontextuelle Datenschutzerklärungen verfolgen einen alternativen Ansatz. Hier werden Datenschutzinformationen für den Nutzungskontext maßgeschneidert und nur die jeweils relevanten Informationen angezeigt. In dieser Arbeit wurde in einer Nutzerstudie untersucht, ob Nutzer diesen Ansatz akzeptieren und ob sie ein besseres, gefühltes Verständnis bezüglich der Datenschutzinformationen haben. Es zeigte sich, dass kontextuelle Datenschutzerklärungen durchweg positiv aufgenommen werden und gegenüber der klassischen Darstellung im Fließtext präferiert werden.

[English translation] Privacy policies are often both hard to find and hard to understand, and because of this, many users read them only partially, or not at all. Contextual privacy policies are a different approach, whereby privacy statements are tailored to the current context of use. Only the respective relevant information is displayed. In this work, a user study was employed to test if users accept this approach, and if they reach a better perceived understanding of privacy policies through it. We could show that contextual privacy policies were received positively, and preferred over the common long textual form of presentation.

Academic Service

Reviewing:

EuroUSEC'24, CHI'24, MuC'23 - Usable Security und Privacy Workshop, CHI'23, CHI'22
Student Volunteer:

MuC'18, MuC'17
Conference Team:

COSIT'19

Teaching

Summer 2023 & 2024

Exercise for Usable Security and Privacy
Summer 2022

Exercise for Usable Security and Privacy (Hybrid)
Summer 2021

Exercise for Usable Security and Privacy (Online)
Summer 2020

Teaching assistant for Survey and Experiment Design (Online)
Winter 2019/20

Teaching assistant for Efficient Data Processing and Storage
Summer 2019 & 2018

Teaching assistant for Survey and Experiment Design
Winter 2017/18

Teaching assistant for User Centered Design and Human Information Behavior
Winter 2016/17

Tutor for Introduction to Computer Science and Media Informatics