Compliance Monitor

• About
• Regulatory Compliance
• Compliance Monitoring Issues
• Business Compliance
• Regulatory vs. Business Compliance
• How To's
• How To: Reproduce Cookie Violations
• Issues / Violations
• Auto Acknowledge
• Issue Handling
• GDPR Export Description
• GDPR Export Description

About

The compliance monitor (codename Krahken) is intended to collect all data from our other products and present them in a unified portal. Additionally it has 2 main purposes: 

• Adding functionality for Compliance Scans and Analytics
• Adding functionality for Asset Management

 

Term description

Term	Description
Asset	Asset can be anything that is scan- and analysable by our crawlers and bots.

Regulatory Compliance

Compliance Monitoring Issues

General Information

In the world of website compliance, a lot of differnt compliance violotions can occour. Therefore we decided to make a clear separation of those violations and introduced different violation categories:

• Regulatory Violations
• Business Violations

Each category includes different types of violations that will be described in the corresponding section. The main differences are that regulatory violatens based on GDPR context prescribed by law and business violations based on custom rules that can be individually defined to meet each customers individual requiremnts.

Regulatory Violations

This category includes all detected problems that relates to the acutal GDPR context prescribed by law. The following types can be detected by the Nimbusec compliance monitor in the context of regulatory violations:

• Cookie Opt-In Generally, the Nimbusec Complaince Scan simulates a standard website user that visits a website, but does not perform any kind of interaction with it (e.g. allowing cookies, ...). For this vioaltion type, we collect all cookie information that was set from the website (before any kind of user interaction). This results in a clear list of cookies that were initially set/provided by the website. In the default configuration, if our simulated website user gets any kind cookies, a 'Cookie Opt-In' violation will be triggerd and shown in the compliance monitor. With other words, cookies have been set without asking for consent in the cookie banner.

In the Nimbusec Compliance Monitor, this violation type can be seen in the compliance view of an asset:

Cookie Issues

By clicking on the "Details" button, all detected cookies can be seen.

If a Cookie Opt-In violation is detected, the following information will be saved.

In more detail, the Nimbusec compliance scan saves the following data:

Example:

Name	Domain	Secure	HttpOnly	LifeTime	SameSite	ThirdParty	InServerResponse
cookie_name	www.example.com	false	false	-1	N/A	true	false

Description of the fields:

Column	Description
Name	Name of the detected Cookie
Domain	Domain name where the described cookie was found
Secure	The Secure flag is an option that can be set by the server when sending a new cookie to the user in an HTTP response. The purpose is to prevent cookies from being observed by unauthorized parties due to sending the cookie in clear text.
HttpOnly	The HttpOnly flag is an additional security option for cookies. If this flag is set, the browser does not display the cookie through client-side scripts.
LifeTime	This parameter shows the validity of a cookie.
SameSite	The SameSite attribute shows if this cookie is restricted to a first-party or same-site context.
ThirdParty	This attribute shows if this cookie was distributed from the own web ressource or from an external one. (domain name matching)
InServerResponse	If the value of this attribute is true, this cookie was directly send from the server. If this value is false, the cookie was created and distributed via a JavaScript context.

• Cookie Banner (ToDo) The Nimbusec Compliance Scan checks websites for the presence of a standard cookie banner. For this case, we check for a default set of various cookie banner implementations. If a custom cookie banner is used, the scan configuration has to be adjusted to detect non default cookie banners.

• Form: HTTP-Transmit For this case, our scan checks all availabe input forms that handles sensitive data. If the content of the input form will be send unencrypted (via HTTP), the Nimbusec Compliance Scan throws a Form: HTTP-Transmit violation. Unencrypted data transmission may be intercepted and the data could be seen in plain text.

By clicking on the "Details" button in the compliance view, all detected cookies can be seen.

Form issues

• Form: Form-Sensitive As mentioned above, the Nimbusec Complaince Scan checks all availabe input forms that handles sensitive data. If we detect such forms, a Form-Sensitive event will be triggered. This should help the users to identify those forms and may handle them in a different way.

• Form: External-Transmit This type of violation gets triggerd if form data will be send to an external processor (e.g. to an external company that may performs user analytics tasks)

Imprint Link Check

• Imprint Policy: The presence of an imprint is a mandaroty regulation in GDPR. Our imprint policy checks the presence of an imprint link on the corresponding website. To detect it, we use predefined search patterns and apply them on the websites source code.

By clicking on the "Details" button in the compliance view, all detected cookies can be seen.

Privacy Policy Link Check

• Privacy Policy: The presence of a privacy statement is a mandaroty regulation in GDPR. Our privacy policy checks the presence of a privacy statement link on the corresponding website. To detect it, we use predefined search patterns and apply them on the websites source code.

Tracker Check

• Tracker: From our point of view, a tracker is any kind of software that collects or transmits data to an external ressource. Therefore, our Nimbusec Compliance Scan analyses the whole network traffic that is active during our simulated website user visits the site. From the GDPR point of view, before the user accepts the defined privacy policy, any kind of data collection or data transmission to third party systems is a violation against the given law.

Business Compliance

Regulatory vs. Business Compliance

When working with the compliance monitor, you will likely very often see the term "regulatory" or "business" in combination with "compliance". 

There is a simple reason for that: Compliance is more complicated than security monitoring. 

While security follows strict rules which count for any website on the internet, compliance is different.

When a website distributes malware, or uses weak or no TLS encryption, it is clearly a security issue. But when a cookie is set without permission, things get a little blurry.

Analysis of compliance topics is a very individual part, where legislation is only a little of a help. We think that the law can be taken into account for any website (in a specific region at least).

TL;DR

Regulatory compliance focusses on checks, which can be performed on almost any websites, because they need to comply with the law. A well known regulation is the general data privacy regulation (GDPR).

Business compliance adds special checks for websites, which need to follow special/individual directives of the company. Therefore they are not required by law, but by internal policies. 

Regulatory Compliance

Regulatory compliance describes rules and patterns, defined by law / legislation of a specific country. An example would be the GDPR for the EU. Per definition, a website accessible to european citizens needs to follow specific rules to protect the privacy of every single person. 

One rule defines, that there must not be set any cookies before the visitor did not give consent. Technical speaking this is a  difficult, as for a lot of web-applications, some cookies are needed to actually let them work. Those are called technical necessary. But this seems still to be a topic for the court. 

Business Compliance

On the other hand we have the business compliance. This describes a set of rules, which can be freely defined by the companies themselves. Those are business decisions which need to be followed to e.g. secure the overall presence on the internet. 

An example would be that the logo of the company should be placed always on the top left of a website. 

Another rule might check if the link to the data privacy page is located on the top AND bottom of the website. 

Also there might be the decision, that specific URLs need to be only accessible from the intranet of the company, and therefore send a statuscode 400 for external visitors. 

Back to cookies: companies want to track their visitors to e.g. measure engagement, or optimize the overall experience for every visitor by studying their movements. Therefore a rule could also check if a specific cookie is set when the website is visited. (This needs of course some very good arguments in the cookie policy of the website, but especially the GDPR got you covered; Predominant legitimate interests (Art. 6 para. 1 lit f)).

How To's

How To: Reproduce Cookie Violations

 

 

This how-to describes how it is best to reproduce cookie violations, as this is often a very confusing topic. First of all it would be good to know, how exactly our crawler works in this case: 

How Nimbusec visits websites

The crawler of Nimbusec is technically based on a headless chrome browser. But it is configured and extended in a way, that it is very hard for website fingerprinting scripts, to detect if this is a BOT (as it is) or a usual visitor. 

EVERY URL IS VISITED FROM A NEW, CLEAN BROWSER WINDOW IN INCOGNITO MODE. NO COOKIES ARE ENABLED, NO PLUGINS OR AD BLOCKER INSTALLED!

Step by step:

1. The crawler receives a base URL to crawl. That can be "http://example.com" or also "http://example.com/foo". 
2. The base URL will be visited as unique user and stored. This one is very important: EVERY URL VISITED, IS COMPARABLE TO A NEW, CLEAN BROWSER WINDOW, without any session cookies or anything else. That means that every page is visited like: 
   1. closing the browser (if any is open currently)
   2. open a new browser in incognito mode
   3. type the whole URL in the address bar and visit it
3. Stored content will be analysed
4. All links will be parsed
   1. external, different to base domain, will be scanned just once for malware and reputation to make sure no harm comes from an outgoing link at the time of scanning
   2. internal links, domain stays the same, will be stored as well for further analysis, and links will be followed
5. Stored content will be analyzed in different flavors - e.g. compliance (including cookies)

How to check this manually

We strongly recommend to do this from within a sandbox, which can be reset to a clean state before proceeding with a new project. This will not only prevent you from being hacked because a website might be attacking, but also working as close to our systems, and most pre-installed user systems as it gets. 

Therefore a virtual machine with a linux would be the best solution. 

Also you need a browser like Chromium, freshly installed, without any cookies.

Requirements

• Virtual Machine with Linux installed
• Browser (e.g. Chromium) without
  • Ad Blocker
  • Configuration to block anything out of the box
• (optional) a VPN tunnel to an Europe location as our scan service takes place from Europe

 

• open a browser in incognito mode

• make sure that blocking mechanisms are deactivated

• open web developer tools

• select the cookie section of the inspection tools

• browse the website to check its cookies

 

Why are there still different results

It may occur that altough all settings are correct and you browse very private to the website, the results are different. We've seen this before when we didn't get the "mf_**" cookie set, because our artificial visitor does not use a mouse, so the tracker did not set the cookie. 

A reason why cookies might look different or aren't set at all, might be the location from where the website is visited. Our crawler performs the compliance scans explicitly from Europe countries. 

That said, if you visit from another country, you might get different results though. The best advice here would be as follows: 

• use a VPN connection to simulate browsing from a different region 
• set the language and country settings of your OS and browser to the VPN location as well

This is of course high profile, and more to configure to reproduce the issue, but it's often worth to verify the results if needed. 

 

Issues / Violations

Auto Acknowledge

Issues can be auto acknowledged to simplify the issue handling process by keeping most of the transparency. 

What does it do?

If there is a rule, that allows e.g. a specific cookie to be set on a website, the issue will be auto acknowledged on our side. This may come handy where a cookie policy already defines cookies that are needed to be set before the visitor of the website accepts the cookie banner. 

So when an asset is analyzed and, taken the example above, some cookies are detected without consent, issues will be created. All issues will be checked if they match the criteria of a specific rule for auto acknowledging. When there is a match, this issue will not pop up in the "pending" section of the violations, but is moved directly to the "done" section. 

It is also tagged as "Auto Acknowledged", and the comment shows the rule that applied to acknowledge this issue. 

Of course, when there already exists an acknowledged issue, we will not create a new one, but update the existing one with the "last seen date". 

How does it work?

Our service team will take requests from customers who wish to setup a specific rule to keep the issue list clean (according to their compliance definitions).

The information needed should be as specific as possible for the rule e.g.:

• All websites should be allowed to set one or all of the following cookies: PHPSESSID, abcd, xyz
• A form asking for "Name" and "Birth date" is allowed on specific websites

Also the websites, where this rule should be applied can be defined very granular by either

• providing a specific list of assets where we should add this rule
• provide a TAG that is set on assets so we can add this rule
• provide other filters we can use to add this rule to various assets

 

Issue Handling

There are 2 types of actions, that can bei set to resolve issues:

• Resolve / Acknowledge
• Ignore

Resolve an issue

 

By resolving an issue, the compliance monitor assumes that the cause has been fixed. 

e.g.: If you resolve an issue for a cookie named "XYZ", we assume that this cookie was removed from the website and will not be detected again. If it is still present on the website, and we detect a cookie "XYZ" again on the next scan of the website, we will open a new issue. 

Why is a new issue opened? 

We create a new issue, because then we have the track record of closed and "reopened" issues. Otherwise it would not be possible to tell, if an issue has been detected and meant to be resolved before. 

The consequence is maybe that there are "duplicates" of issues. 

How to resolve an issue?

An issue can be resolved by simply using the "Check" button aside the issue entry. This will open up a modal form (title: "Close issue") where it is possible to add a comment for future reference. 

e.g.: if you ignore a cookie-issue named "ABC", the system will not create a new issue if a cookie with the name "ABC" is detected again on this website.

After committing with the "Close" button, the issue is marked as resolved and moved to the "Done" Tab. 

Alongside with the comment we also store the userID and date when the action happened. That allows further transparency from whom the issue was closed through the compliance monitoring portal.

* Use the check-button to perform an action on an issue

Ignore an issue

When ignoring an issue, the portal assumes that this issue is intended and, in compliance terms, compliant with either the company guidelines and or regulatory guides. 

Therefore when we detect an ignored issue again on a website, we will not open a new issue, because it was ignored in the past. The comment feature can be used to take a note on why this decision was made. With any action performed on an issue, the userID and date will be stored, so it can be tracked who decided to close the issue. 

How to ignore an issue

Ignoring an issue is as simple as resolving it and follows the same principle. Just use the check-button aside the issue entry. When the modal to close the issue pops up, don't forget to tick the checkbox for ignoring an issue. 

* to ignore an issue, just activate the checkbox in the closing modal to do so.

 

 

GDPR Export Description

GDPR Export Description

General Information

The GDPR export contains all collected information/results of the performed GDPR scan(same information as in the Nimbusec compliance monitor).
Information is split in 4 main categories

• Inputs
• Cookies
• Tracker
• [Content] (optional)

All collected information/results were generated before any kind of user interaction with the website was performed  No cookies etc. were accepted when visited the website[except the cookie banner feature was enabled (optional)].

The Excel Document itself contains 5 different data sheets

• Input
• Cookies
• Tracker
• Content (Optional)
• Assets

The "Assets" sheet contains all websites that were scanned for compliance.

Inputs

This section focuses on input forms that handle sensitive data. Our scanner collects all form fields that are present on your web-applications.
Simply put: A complete inventory of all input forms on all domains in scope.

Table fields explained

• Date
  Shows the date when this information was discovered
• Asset
  Indicates the start point that was scanned
• Schema
  Which HTTP schema was used for the scan (HTTP/HTTPS)
• URL
  Shows the exact URL of the detected input form
• Art 9/10 Category
  True shows a violation against this specific case
• Input Name
  Name of the detected input field (parsed from source code)
• Input Category
  Based on the detected input name and input type, an input category is defined Contact data, job, gender, user data, …
• Input ID
  Represents the unique ID of the input field (parsed from source code)
• Input Type
  Shows the type of the detected input field  e.g. text, radio button, select, …
• Placeholder
  Shows, if present, the prepared default text
• Form Target Schema
  Shows the used schema that was used to transmit the provided data
  https -> data encrypted
  potentially http -> transmission schema unsure, please check manually (with e.g. Wireshark
  http -> data unencrypted
• Form Action
  Provides the defined action for the specific form (from source code)

Next steps based on best practices (work package)

• Remove / Update unencrypted forms  https
• Individual website review
• Check if forms are still necessary
• Check if placed input data is needed in the placed context
• Check if provided data is processed / sent outside your organization (third party)
• Check if all provided form data is documented in the privacy policy including the reason why this data is needed

Further Reading

Unencrypted data forms
https://eur-lex.europa.eu/legal-content/DE/TXT/HTML/?uri=CELEX:32016R0679&from=EN#d1e3385-1-1
https://www.mein-datenschutzbeauftragter.de/blog/datenschutz-auf-websites-warum-die-verschluesselung-von-kontaktformularen-etc-wichtig-ist/
https://gdpr-info.eu/issues/encryption/
https://www.ra-plutte.de/gastbeitrag-warum-sie-ihre-website-auf-https-umstellen-sollten/

Personal Identifiable Information (PII)
https://eur-lex.europa.eu/legal-content/DE/TXT/HTML/?uri=CELEX:32016R0679&from=EN#d1e2066-1-1

Third Parties
https://www.usp.gv.at/it-geistiges-eigentum/datenschutz/einwilligung.html
https://dsgvo-gesetz.de/themen/auftragsverarbeitung/
https://www.lda.bayern.de/media/veroeffentlichungen/FAQ_Abgrenzung_Auftragsverarbeitung.pdf

Cookies

This section focus on detected cookies of the scanned websites. Since our scanner does not interact with the website, all detected cookies were set before user consent.
-> A full documentation of all set cookies (before user consent) of all scanned web-applications.

Table fields explained

• Date
  Shows the date when this information was discovered
• Asset
  Indicates the start point that was scanned
• Schema
  Which HTTP schema was used for the scan (HTTP/HTTPS)
• URL
  Shows the exact URL of the detected cookieIn case of a manual review, please check this URL
• Name
  Includes the name of the detected cookie
• Value
  Shows the set value of the detected cookie
• Domain
  Shows the domain that sets the detected cookie
• ThirdParty
  Has a direct relation to the Domain column, if the Asset and the Domain is not equal (based on domain level) a third party sets this cookie

Next steps based on best practices (work package)

• Check all detected cookies and classify them as “technically necessary” or not (depends on the application)
• Implement/Update a cookie banner to ensure that all ”non technically necessary” cookies were set after user consent
• Create/Update privacy policy / cookie policy and list all used cookies

Further Reading

https://curia.europa.eu/juris/document/document.jsf;jsessionid=F2A804042CAC4FE3D70A00596C6A76D0?text=&docid=218462&pageIndex=0&doclang=DE&mode=req&dir=&occ=first&part=1&cid=1686588
https://www.wbs-law.de/it-und-internet-recht/datenschutzrecht/eugh-cookies-aktive-einwilligung-c-673-17-45473/
https://www.wko.at/branchen/information-consulting/werbung-marktkommunikation/eugh-entscheidung-zu-cookies-und-einwilligung.html
https://www.datenschutz.org/cookies/#die-regelungen-fuer-cookies-innerhalb-der-eu
https://www.lda.bayern.de/media/pm/pm2021_06.pdf

Tracker

This section focuses on internal/external used tracking software. Our scanner analyses the whole network traffic that was triggered by initially visiting the web-application. As a result, all detected requests were triggered before user consent.
-> A complete list of all used tracking technologies of each domain in scope

Table fields explained

• Date
  Shows the date when this information was discovered
• Asset
  Indicates the start point that was scanned
• Schema
  Which HTTP schema was used for the scan (HTTP/HTTPS)
• Category
  All detected tracker will be assigned to a specific category e.g. FingerPrinting, Social Media, …
• Date
  Shows the date when this information was discovered
• Asset
  Indicates the start point that was scanned
• Schema
  Which HTTP schema was used for the scan (HTTP/HTTPS)
• Category
  All detected tracker will be assigned to a specific category e.g. FingerPrinting, Social Media, …

Next steps based on best practices (work package)

• Disable/Remove all events (that process personal user data) that were triggered by initially visiting the website (before user consent)
• Implement (if not present) a user consent banner (e.g. cookie banner)
• Create/Update privacy policy (https://gdpr.eu/privacy-notice/)

Further Reading

https://dsgvo-gesetz.de/themen/auftragsverarbeitung/
https://gdpr.eu/privacy-notice/

Content

This section focus on company specific compliance parts that were individually defined with the project team.
->A full documentation of which domain violates the defined rules

Table fields explained

• Date
  Shows the date when this information was discovered
• Asset
  Indicates the start point that was scanned
• Schema
  Which HTTP schema was used for the scan (HTTP/HTTPS)
• URL
  Shows the exact URL of the checked ressource
• OK
  This flag shows if the rule (in the Rule column) is was violated or not
  true = no violation detected
  false = violation detected
• Rule
  Shows the rule that was checked for the given asset

Next steps based on best practices (work package)

• Analyze individual rule violations of each website(Why was this part not detected?)
• Apply changes to the web-application to meet the company rules