Driving decision making with custom datasets

Atlas is dedicated to complying with all different legal data protection requirements and guidelines as defined by the GDPR, CCPA, and SOC 2. We take great effort to ensure that at every layer we are following these privacy laws.

Regarding GDPR, please be mindful that GDPR’s "Right to be forgotten" provision of the GDPR regulation may affect results extracted from EU-based search engine results but it will not affect results shown through U.S.-based search engines which are protected under the U.S. Constitution’s First Amendment.

Regarding CCPA, Atlas does not sell any of our client’s personal information as outlined by the CCPA in any manner.

Regarding SOC 2 compliance, Atlas is committed to following the 5 service criteria as defined by SOC 2. Atlas has a strong emphasis on cybersecurity by following best practices such as MFA, PoLP, encryption at rest, and secure password sharing.

Web scraping is legal and protected by U.S. law provided it respects certain use cases.

Regarding legality, we will not service any illegal requests including but not limited to launching distributed denial of service attacks (DDOS), terrorism, website hacking, or using extracted data to violate copyright laws.

In addition, we follow the privacy policies and guidelines as outlined by the Computer Fraud and Abuse Act (CFAA), General Data Protection Regulation (GDPR), California Consumer Privacy Act (CCPA), and System and Organization Controls 2 guidelines (SOC 2).

With regard to copyright law, web scraping is effectively treated as an automated form of manual data collection. As such, we will not service web scraping requests intended to violate copyright laws protecting content created on the internet.

Lastly, alongside legal regulations and guidelines, we adhere to ethical data scraping practices that guide the scale, speed, and usage of how we conduct scrapes. This includes but is not limited to hard limiting concurrent thread count and running code at low-traffic periods to minimize target server bandwidth usage.

Web scraping is the process of writing code to automate and extract data at scale from the web. It is the bedrock technology that powers tools such as Google Search and OpenAI’s ChatGPT. The web scraping pipeline consists of sending outbound HTTP requests, analyzing returned HTML/JSON content, parsing relevant portions, and storing the data to be then passed downstream into data engineering pipelines.

Atlas custom builds bots for every website to ensure the highest level of granularity from the data extraction. Atlas creates unique mapping strategies for each website and parameterizes each bot with its own set of request headers, cookies, fingerprints, IP addresses, and more. Following this approach allows us to perform much larger and more detailed mappings of websites.

We make great efforts to fully adhere to privacy regulations and fair usage as defined by different legal rulings and court cases. Along with this, we follow ethical guidelines for running web extractions, including things such as hard limiting concurrent thread count and running code at low-traffic times in order to minimize the target server bandwidth usage.

Atlas is committed to handling everything that falls under the data engineering lifecycle. From sourcing, cleaning, extracting, and learning from data, Atlas provides a blend of the following services with each project:

Data Scraping: Our primary method of sourcing custom data for your business
‍
Data Engineering: Cleaning and harmonizing raw web data across websites to be more digestible for downstream analysis
‍
Data Entry: Augment datasets with custom-assigned labels tagged by our data entry teams
‍
Data Visualization: Create stellar dashboards and charts in the BI tool of your choice
‍
Artificial Intelligence: Train and fine-tune custom AI models
‍
Data Analytics & Report Generation: Generate easy-to-share CSV/Excel reports
‍
Consultation: We offer consultation services to help scope out large projects and answer any more detailed questions relating to our process and project feasibility.

Lastly, we handle all aspects of integration with proxy servers, cloud computing resources, and database design internally.

At Atlas, we primarily work on a fixed-price basis. We will start by understanding your full project needs and what deliverables needs to be created in order to consider the project a win for you. After clearly defining the various project goals and milestones, we will share a fixed price estimate for all components of the project roadmap.

At Atlas, we have worked on dozens of small, medium, and enterprise-level contracts ranging in cost from $2.5K - $6.8K, up to $59.5K. We also offer add-on hourly work for additional ad-hoc requests needed to finalize the deliverable.

Depending on how large the project is, Atlas will provide a custom estimate based on the project's demands. Shoot us a message if you are interested in discussing a project together!

GoogleBots crawl website URLs in order to create a ranked index that is accessible through Google Search. GoogleBots are highly generic crawling software that follows a very similar approach for every website architecture, pulling down each website’s text without much regard to the target website's structure.

On the other hand, AtlasBots deploy a unique AtlasBot handcrafted for the different web architectures of a given website. These individual AtlasBots are then further customized for sub-page web architectures found within large websites - such as creating an AtlasBot for YouTube’s search feed, an AtlasBot for YouTube’s channel pages, and an AtlasBot for YouTube’s video pages. AtlasBots are highly specialized for every component in a target website to ensure the widest and deepest mapping of the website is reached.

GoogleBots optimize primarily for text extraction and page indexing in order to create a ranking, whereas AtlasBots optimize for breadth and depth of data extraction in order to create detailed analytic reports.