Thursday, 30 April 2020

Faster SQL Queries on Delta Lake with Dynamic File Pruning

There are two time-honored optimization techniques for making queries run faster in data systems: process data at a faster rate or simply process less data by skipping non-relevant data. This blog post introduces Dynamic File Pruning (DFP), a new data-skipping technique enabled by default in Databricks Runtime 6.1, which can significantly improve queries with selective joins on non-partition columns on tables in Delta Lake.

In our experiments using TPC-DS data and queries with Dynamic File Pruning, we observed up to an 8x speedup in query performance and 36 queries had a 2x or larger speedup.

In experiments using TPC-DS data and queries with Dynamic File Pruning, Databricks observed up to an 8x speedup in query performance and 36 queries had a 2x or larger speedup.

The Benefits of Dynamic File Pruning

Data engineers frequently choose a partitioning strategy for large Delta Lake tables that allows the queries and jobs accessing those tables to skip considerable amounts of data thus significantly speeding up query execution times. Partition pruning can take place at query compilation time when queries include an explicit literal predicate on the partition key column or it can take place at runtime via Dynamic Partition Pruning.

Delta Lake on Databricks Performance Tuning

In addition to eliminating data at partition granularity, Delta Lake on Databricks dynamically skips unnecessary files when possible. This can be achieved because Delta Lake automatically collects metadata about data files managed by Delta Lake and so, data can be skipped without data file access. Prior to Dynamic File Pruning, file pruning only took place when queries contained a literal value in the predicate but now this works for both literal filters as well as join filters. This means that Dynamic File Pruning now allows star schema queries to take advantage of data skipping at file granularity.

Per Partition Per File (Delta Lake on Databricks only)
Static (based on filters) Partition Pruning File Pruning
Dynamic (based on joins) Dynamic Partition Pruning Dynamic File Pruning (NEW!)

How Does Dynamic File Pruning Work?

Before we dive into the details of how Dynamic File Pruning works, let’s briefly present how file pruning works with literal predicates.

Example 1 – Static File Pruning

For simplicity, let’s consider the following query derived from the TPC-DS schema to explain how file pruning can reduce the size of the SCAN operation.

    -- Q1
    SELECT sum(ss_quantity) 
    FROM store_sales 
    WHERE ss_item_sk IN (40, 41, 42) 

Delta Lake stores the minimum and maximum values for each column on a per file basis. Therefore, files in which the filtered values (40, 41, 42) fall outside the min-max range of the ss_item_sk column can be skipped entirely. We can reduce the length of value ranges per file by using data clustering techniques such as Z-Ordering. This is very attractive for Dynamic File Pruning because having tighter ranges per file results in better skipping effectiveness. Therefore, we have Z-ordered the store_sales table by the ss_item_sk column.

In query Q1 the predicate pushdown takes place and thus file pruning happens as a metadata-operation as part of the SCAN operator but is also followed by a FILTER operation to remove any remaining non-matching rows.

In experiments using TPC-DS data and queries with Dynamic File Pruning, Databricks observed up to an 8x speedup in query performance and 36 queries had a 2x or larger speedup.

When the filter contains literal predicates, the query compiler can embed these literal values in the query plan. However, when predicates are specified as part of a join, as is commonly found in most data warehouse queries (e.g., star schema join), a different approach is needed. In such cases, the join filters on the fact table are unknown at query compilation time.

Example 2 – Star Schema Join without DFP

Below is an example of a query with a typical star schema join.

    -- Q2 
    SELECT sum(ss_quantity) 
    FROM store_sales 
    JOIN item ON ss_item_sk = i_item_sk
    WHERE i_item_id = 'AAAAAAAAICAAAAAA'

Query Q2 returns the same results as Q1, however, it specifies the predicate on the dimension table (item), not the fact table (store_sales). This means that filtering of rows for store_sales would typically be done as part of the JOIN operation since the values of ss_item_sk are not known until after the SCAN and FILTER operations take place on the item table.

Below is a logical query execution plan for Q2.

Example query where filtering of rows for store_sales would typically be done as part of the JOIN operation since the values of ss_item_sk are not known until after the SCAN and FILTER operations take place on the item table.

As you can see in the query plan for Q2, only 48K rows meet the JOIN criteria yet over 8.6B records had to be read from the store_sales table. This means that the query runtime can be significantly reduced as well as the amount of data scanned if there was a way to push down the JOIN filter into the SCAN of store_sales.

Example 3 – Star Schema Join with Dynamic File Pruning

If we take Q2 and enable Dynamic File Pruning we can see that a dynamic filter is created from the build side of the join and passed into the SCAN operation for store_sales. The below logical plan diagram represents this optimization.

Example query with Dynamic File Pruning enabled, where a dynamic filter is created from the build side of the join and passed into the SCAN operation for store_sales.

The result of applying Dynamic File Pruning in the SCAN operation for store_sales is that the number of scanned rows has been reduced from 8.6 billion to 66 million rows. Whereas the improvement is significant, we still read more data than needed because DFP operates at the granularity of files instead of rows.

We can observe the impact of Dynamic File Pruning by looking at the DAG from the Spark UI (snippets below) for this query and expanding the SCAN operation for the store_sales table. In particular, using Dynamic File Pruning in this query eliminates more than 99% of the input data which improves the query runtime from 10s to less than 1s.

Scan node statistics , demonstrating the effect of dynamic file pruning on query performance.

Without dynamic file pruning

Scan node statistics , demonstrating the effect of dynamic file pruning on query performance.

With dynamic file pruning

Enabling Dynamic File Pruning

DFP is automatically enabled in Databricks Runtime 6.1 and higher, and applies if a query meets the following criteria:

  • The inner table (probe side) being joined is in Delta Lake format
  • The join type is INNER or LEFT-SEMI
  • The join strategy is BROADCAST HASH JOIN
  • The number of files in the inner table is greater than the value for spark.databricks.optimizer.deltaTableFilesThreshold

DFP can be controlled by the following configuration parameters:

  • spark.databricks.optimizer.dynamicPartitionPruning (default is true) is the main flag that enables the optimizer to push down DFP filters.
  • spark.databricks.optimizer.deltaTableSizeThreshold (default is 10GB) This parameter represents the minimum size in bytes of the Delta table on the probe side of the join required to trigger dynamic file pruning.
  • spark.databricks.optimizer.deltaTableFilesThreshold (default is 1000) This parameter represents the number of files of the Delta table on the probe side of the join required to trigger dynamic file pruning.

Note: In the experiments reported in this article we set spark.databricks.optimizer.deltaTableFilesThreshold to 100 in order to trigger DFP because the store_sales table has less than 1000 files

Experiments and Results with TPC-DS

To understand the impact of Dynamic File Pruning on SQL workloads we compared the performance of TPC-DS queries on unpartitioned schemas from a 1TB dataset. We used Z-Ordering to cluster the joined fact tables on the date and item key columns. DFP delivers good performance in nearly every query. In 36 out of 103 queries we observed a speedup of over 2x with the largest speedup achieved for a single query of roughly 8x. The chart below highlights the impact of DFP by showing the top 10 most improved queries.

Dynamic File Pruning reduces by a large factor the number of files read in several TPC-DS queries running on a 1TB dataset.

Many TPC-DS queries use a typical star schema join between a date dimension table and a fact table (or multiple fact tables) to filter date ranges which makes it a great workload to showcase the impact of DFP. The data presented in the above chart explains why DFP is so effective for this set of queries — they are now able to reduce a significant amount of data read. Each query has a join filter on the fact tables limiting the period of time to a range between 30 and 90 days (fact tables store 5 years of data). DFP is very attractive for this workload as some of the queries may access up to three fact tables.

Getting Started with Dynamic File Pruning

Dynamic File Pruning (DFP), a new feature in Databricks Runtime 6.1, can significantly improve the performance of many queries on Delta Lake. DFP is especially efficient when running join queries on non-partitioned tables. The better performance provided by DFP is often correlated to the clustering of data and so, users may consider using Z-Ordering to maximize the benefit of DFP. To leverage these latest performance optimizations, sign up for a Databricks account today!

--

Try Databricks for free. Get started today.

The post Faster SQL Queries on Delta Lake with Dynamic File Pruning appeared first on Databricks.

Call for User Stories - Jenkins is the Way

Jenkins Is The Way

One of the things we loved about going to developer conferences was meeting Jenkins users — newbies and old-timers alike — who are excited to talk about their projects and share tips on how to move forward using Jenkins. Since the coronavirus pandemic, we’re learning to rely more on new ways to gather, and it’s happening via Jenkins online meetups, GitHub collaborations, and Twitter threads, to name a few.

It’s a significant change. But what hasn’t changed is the need to share stories about the things users have built, the solutions they’ve developed, and the excellent results they’re getting from some really innovative Jenkins implementations. Then we wondered, why isn’t anyone collecting these user stories and sharing them with the Jenkins community.

Introducing Jenkins is the Way

So we took the first step to record and archive all the great stuff everyone in our community is building with Jenkins. This way, Jenkins users old and new can come to an archive and search for Jenkins solutions for inspiration. We foresee a vast library of solutions from all around the world, solving a wide array of challenges in every industry imaginable. We decided to call this archive "Jenkins Is The Way" and host it at https://JenkinsIsTheWay.io .

To aggregate all these stories, we built a simple online questionnaire so that Jenkins users can submit their own experience using this leading open source automation server. With so many plugins to support building, deploying, and automating your projects, we expect to see a vast collection of stories.

We’ve already received a handful, including stories that illustrate how Jenkins Is The Way:

Add your story. Show your Jenkins pride. Get our T-shirt

Jenkins Is The Way T-shirt

Be an inspiration to the Jenkins community by sharing your Jenkins story. Just go to this link and fill out the form. We’ll ask you about your project’s goals, the technical challenges you overcame with Jenkins, and the solutions you created. It should take no more than 20-30 minutes to complete.

We’ll clean it up for clarity and publish it on https://JenkinsIsTheWay.io .

Once it’s part of our archive, we’ll send you our new 2020 Jenkins Is the Way t-shirt.

And since the more, the merrier, please share this blog post with peers and colleagues. We want to hear everyone’s stories about the clever ways Jenkins is used to automate all that we need to do.

Thanks and Acknowledgement

Special thanks to abConsulting for creating and managing the https://JenkinsIsTheWay.io site and for reviewing, editing, and publishing the submitted stories.

Thanks to the Jenkins Advocacy and Outreach SIG for their reviews and feedback.

Thanks also to CloudBees for sponsoring the "Jenkins is the Way" program.

CloudBees

Unbelievable impact of COVID-19 on IT outsourcing!

IT Outsourcing is one of the most flourishing fields in the world. IT outsourcing is nothing new, but it is continuously growing. More and more businesses have realized the importance of outsourcing and have consulted various outsourcing companies for the services. However, amidst this tough situation, the pandemic, the IT outsourcing industry has been affected as well. COVID-19 has impacted almost everything in the world, including the IT outsourcing industry. However, the impact isn’t as predictable or expected as it was supposed to be. In some places, the impact has been severe, however, in some areas, the impact hasn’t been much. It depends on a lot of factors. In this article, we will discuss a few things that have impacted the industry, both positively and negatively. The positive impact First, let’s talk about the positive impact of IT Outsourcing. The world is accepting the new way of working, and therefore, a lot of demand for the IT industry. IT industry has been working tirelessly to fulfill the demands of various businesses. In fact, many companies have started opting for more services from the IT Outsourcing service providers, considering the current situation. Also, many businesses, who have never consulted any IT ...


Read More on Datafloq

Wednesday, 29 April 2020

Universities turn to 3D printing for Covid-19 gear

Several universities and institutions across India are now using 3D printers to make face shields as well as ventilator splitters

Evolving the Databricks brand

.

Some brands start out as, well, brands. A lot of work goes into the concept and painting the picture before the business is ever launched.

Databricks is different. It always has been and always will be an engineering-led company.

Databricks’ model for innovation is inspired by the open-source community. This is where our roots run deepest, as it underpins everything that makes us special — our platform, our culture and our ethos. And like the open-source community, Databricks is driven by the spirit of collaboration and its impact on innovation.

There weren’t many people willing to bet that a bunch of students and teachers in a research lab at Berkeley — with virtually no business experience among them — could take the open-source software that they helped create, enhance it and deliver it as a cloud-based platform for data and AI. Three major technology transformations would have to unfold in order for this to succeed:

  1. The cloud would have to go mainstream
  2. Open source software would have to become a standard in the enterprise
  3. Machine learning would have to become more than science fiction

Today, Databricks is one of, if not the fastest growing SaaS company in history. Our success is attributable to our relentless focus on innovation and making customers successful. More specifically, to putting all of our energy into simplifying data and AI so that data teams – engineers, scientists and analysts – can innovate faster.

But we didn’t get here because we focused on building an amazing brand.

In fact, some might argue that we intentionally avoided the “B word” because it was too much of a distraction. Some might say that focusing on it would have led to some kind of radical reinvention that would make us lose who we really were. Some might say that the absence of a brand was Databricks’ brand. But the truth is our brand was always there, embedded in our core values and the way we approached everything we do.

Brand is about how you’re perceived – it’s not something you can define for yourself. It’s crafted over time, through the sum of experiences that people have with your company. By definition, a brand needs this time — about seven years in our case — before it can really be captured and defined in a way that’s credible and true to your roots.

This is the first time we’ve ever been intentional about capturing our brand. We spent a lot of time with employees, our customers, developers and partners to understand how the people who know us best think about Databricks. Three consistent themes emerged, each of which describe how we operate internally as well as how our customers think about us:

Collaboration – We believe innovation happens faster when we work together, learn from each other, iterate and constantly improve. More than ever, data and AI is a team sport and Databricks is the platform for data teams.

Innovation – We believe in science and the limitless potential of data. Databricks enables organizations to realize that potential as quickly as possible.

Impact – Impact is everything. It’s why people work at Databricks. It’s what drives us. Most importantly, it’s what organizations depend on us for. We live in awe of the impact our customers are having on the world and are inspired by the role that Databricks plays.

We put a lot of thought into how we wanted to represent these attributes, both in our look and our language. Needless to say, we’re proud of the result — not just because it looks good or sounds good — but because it’s authentic to how people think about Databricks and the value that we bring to the table.

We think the video at the top of this blog captures it well – who we are, what we do and what our customers and the entire data community can achieve on an open, unified platform for data and AI. We’d love to hear what you think – email us at brand@databricks.com.

We didn’t always use these words to say it, but our mission has always been to help data teams solve the world’s toughest problems. Never have we been more proud or felt such a sense of urgency for what we can do together with our customers and partners. And as our mission continues to unfold, we can’t wait to see how our brand continues to evolve. We are just getting started!

--

Try Databricks for free. Get started today.

The post Evolving the Databricks brand appeared first on Databricks.

A closer look at the role of big data during the COVID-19 crisis

Throughout human history, it is safe to say that we have put the planet through more than our fair share of grief and turmoil. Through unprecedented revolutions set in stone with the aim of furthering our own progress and longevity as a dominant species, we have changed the world time and again. Some of these changes have proven to be positive, while others have proven to be quite the opposite. It all depends not on the intent of the initial actions that spur on the evolution, but on the consequences, on how the chips fall. In recent months, we have seen one of the worst evolutions in a long time. In fact, for many of us, this is the first (and hopefully the last) time we have lived to see a pandemic rise and play out around the globe. The ongoing now-global Coronavirus pandemic has drastically revolutionised the world as we know it in just about every possible way. Believe it or not, we can become more digitally and technologically inclined (or obsessed, if you will). We have never seen innovation happen so quickly, such as the rapid development of COVID-19 test kits such as the A*STAR Fortitude Kit 2.0.As ...


Read More on Datafloq

Blockchain Technology Could Disrupt The Insurance Industry

As technology advances, traditional financial assets like insurance policies will have to change if they want to compete in our world. Other financial sectors have embraced blockchain technology and cryptocurrencies while insurance lags behind — leaving the industry saddled with expensive, inefficient processes.Customers and clients in the insurance world want to know how blockchain technology can improve their experience with insurance. Does it get cheaper? Faster? Will this make car insurance companies non-existent? Here’s what you need to know.Blockchain and insuranceInsurance processes are very antiquated. Paper contracts and brokers acting as middlemen are just two examples of how the insurance industry is stuck in the past.Every extra step in the process is another place where information can be tampered with or lost. When contracts are processed on paper, unnecessary steps are taken to set up a policy; this leaves more room for error and fraud. The FBI estimates that fraud accounts for more than $40 billion a year in the insurance industry, which adds up to $400 a year to the average person's insurance rates.According to IBM, there are three places where the insurance industry could see significant benefits from blockchain technology:Client Onboarding: It allows for secure sharing of information ...


Read More on Datafloq

State-sponsored hackers circle around govt's Covid-19 efforts

Cybersecurity firm Cyfirma alerts India's nodal cyber security agency Cert-In about Dark Web conversations of a Pakistan state-sponsored group

Tuesday, 28 April 2020

Why Tech Needs A Brand Strategy

In the tech industry, not many businesses consider their brand strategy too profoundly. This failing means that instead of standing out in an already saturated field, they just form part of the background. A brand strategy focuses on promoting the company's image to the consumer. As The Balance SMB notes, a brand is an item or characteristic that marks a particular business as unique. A brand doesn't necessarily have to be as recognizable as Dermani Medspa, for example, but it helps to connect to clients when they are. Tech companies need a brand strategy because they're trying to connect their userbase to their product. They're trying to leverage their core audience's loyalty to market their services. Doing so requires breaking down the elements of the brand's image.Getting the Value Proposition RightOne of the things that marketing students learn early on is how important branding is for a company's success. Information Week notes a concept known as functional branding, whereby a tech company uses every opportunity afforded to them to build customer loyalty. Branding is more than just a fancy logo, and a catchy motto (although these too form part of the strategy). Functional branding offers a value promise to the ...


Read More on Datafloq

New study: Databricks delivers nearly $29 million in economic benefits and pays for itself in less than six months

New commissioned study by Forrester Consulting on behalf of Databricks finds that Databricks customers experience revenue acceleration, improved data team productivity and infrastructure savings on their data analytics and AI projects

A new commissioned study by Forrester Consulting on behalf of Databricks finds that Databricks customers experience revenue acceleration, improved data team productivity and infrastructure savings resulting in a 417% ROI on their data analytics and AI projects.

According to Forrester, “In today’s hypercompetitive business environment, harnessing and applying data, business analytics, and machine learning at every opportunity to differentiate products and customer experiences is fast becoming a prerequisite for success.’’ So it’s no wonder enterprises are betting big on data analytics and AI. In fact, roughly 65% of CIOs at Fortune 1000 companies plan to invest over $50 million in data and AI projects in 2020.

But not all data and AI strategies are created equal. Recent studies have shown that only one in three data and AI projects are successful. This is largely a result of legacy analytics investments that lack the scale, collaboration features, and modern big data and AI capabilities required to build and deploy advanced analytics products. With organizations struggling to deliver data-driven innovation, this begs the question: What enterprise-grade technologies should organizations invest in to help their data teams be successful? And how can organizations quantify the impact of these investments?

Delivering measurable business value with Databricks

To help answer these questions, Databricks commissioned a Forrester Consulting study: The Total Economic Impact™ (TEI) of the Databricks Unified Data Analytics Platform. In this new study, Forrester examines how data teams — and the entire business — can move faster, collaborate better and operate more efficiently when they have a unified, open platform for data engineering, machine learning, and big data analytics. Through customer interviews, Forrester found that organizations deploying Databricks realize nearly $29 million in total economic benefits and a return on investment of 417% over a three-year period. They also concluded that the Databricks platform pays for itself in less than six months.
 
Databricks delivers business value
 

More specifically, data teams interviewed for the study experienced the following key benefits from the Databricks Unified Data Analytics Platform:

Increased revenues by accelerating data science outcomes

Databricks customers achieved a 5% increase in revenues by enabling data science teams to build more — and better — ML models, faster. Additionally, Databricks democratized data access across the organization. This led to new users creating a diverse set of new analytics products such as recommendation engines, pricing optimizations and predictive maintenance models. All these innovations led to top-line growth.

“With Databricks, we are able to train models against all our data more quickly, resulting in more accurate pricing predictions that have had a material impact on revenue.” – Bryn Clark, Data Scientist, Nationwide

Read Nationwide’s story

Improved productivity of data teams

Databricks improved customer productivity of data scientists and data engineers by 25% and 20%, respectively. Customers shared that the improved data management capabilities enabled data teams to spend less time searching for and cleaning data, less time creating and maintaining ETL pipelines, and more time building analytics and ML models to drive meaningful business outcomes. Databricks also helped remove technical barriers that limited collaboration among analysts, data scientists, and data engineers, enabling data teams to work together more efficiently.

“Being on the Databricks platform has allowed our team of data scientists to make huge strides in setting aside all those configuration headaches that we were faced with. It’s dramatically improved our productivity.” – Josh McNutt, SVP of Data Strategy and Consumer Analytics, Showtime

Read Showtime’s story

Significant cost savings retiring legacy analytics platforms

By migrating to Databricks, interviewed organizations were able to retire on-premises infrastructure and cancel legacy software licenses, resulting in millions of dollars of savings. Additionally, the management of the Databricks platform proved substantially easier than legacy environments. This enabled customers to reallocate IT resources to higher-value projects and reduce operational costs.

“Databricks has enabled Comcast to process petabytes of data while reducing compute costs by 10x . Teams can spend more time on analytics and less time on infrastructure management.”

Read Comcast’s Story 

Read the Forrester TEI study

With the Databricks Unified Data Analytics Platform, customers can now accelerate data-driven innovation, thanks to a unified, open platform for data science, ML, and analytics that brings together data teams, processes, and technologies.

To find out more, download the full Total Economic Impact study for Databricks.

DOWNLOAD NOW!

--

Try Databricks for free. Get started today.

The post New study: Databricks delivers nearly $29 million in economic benefits and pays for itself in less than six months appeared first on Databricks.

Why You Need File Transfer Governance

The onslaught of data security breaches today is relentless, with thousands of major breaches each year and 50 percent more breaches in 2019 vs. 2018, according to a report by Risk Based Security. The costs for each breach have burgeoned as well, with the average cost of a data breach at about $3.92 million.Securing data from breaches not only spares bottom line and publicity, it's now also a basic legal requirement to comply with rapidly growing data privacy laws.While organizations have long had to comply with industry-specific standards, such as HIPAA in healthcare and the Payment Card Industry Data Security Standard (PCI DSS), you now also face new consumer privacy regulations. Including:GDPR from the European UnionCalifornia Consumer Privacy Act (CCPA)Canada's Personal Information Protection and Electronic Documents Act (PIPEDA).Protecting data starts with data governance, the process of creating and enforcing rules & policies to ensure information is formally and properly managed throughout the enterprise. One often overlooked but critical element of data governance: file transfer governance.What is File Transfer Governance?When organizations consider data governance, they typically think about data sitting in their database, data warehouse, and applications. They often overlook file transfer governance, or the governance of data in motion. But ...


Read More on Datafloq

Companies bet on AI cameras to track social distancing, limit liability

It will allow companies to show not only workers and customers, but also insurers and regulators, that they are monitoring and enforcing safe practices.

How Big Data Is Impacting Healthcare

Big Data has filled an essential role in healthcare. It's immediately apparent applications as a means of storing and recalling patient medical history pales in comparison to what else that data can be used for. Big Data, from an enterprise perspective, deals with the collection of multiple streams of information from various sources. These data packets are collected and processed (usually in real-time) to offer insights based on that data. However, patient medical data is confidential, or at least it should be. As The Guardian informs us, the total collected worth of the British NHS's client data is in the billions. It raises the questions as to whether utilizing Big Data is more of a curse than a blessing.Who Has Your Data?Big Data is a double-edged sword for the healthcare industry. Not only is patient data a sensitive issue, but recent leaks in other sectors have raised doubts in the safety of user data overall. According to Forbes, in July 2019, the British Parliament agreed with Amazon to hand over NHS healthcare information to make searching for symptoms easier on non-specialists. This caused a massive uproar, as users had no say in what happened with their personal data. While it's ...


Read More on Datafloq

Monday, 27 April 2020

From Packaging to Delivery: How 3D Design is Essential for Supply Chain

The worlds of business and retail never stay still for very long. New innovations and technologies appear that can turn entire industries on their head in a moment and if you are not paying attention to them, you can quickly get left behind by your rivals and competitors.3D printing is one of these new technologies looking to revolutionise the sector and it will affect every industry around the world. However, one of it’s more significant effects is that it is currently playing a key role in transforming the supply chain. It is impacting every stage of the product development lifecycle, from the inception of the product itself to the very nature of the packaging, presentation and delivery processes associated with it. Firstly, if you are unaware, 3D printing is a process of using a three-dimensional digital model to create a physical object by adding many thin layers of material in succession. So let’s take a look at how 3D printing affects supply chain and just what is the role of 3D printing in SCM?How 3D Design is Revolutionising The Supply Chain


What ...


Read More on Datafloq

Types of Backups That You Can Have with Dedicated Servers

Understanding Dedicated ServersWhen we consider the web hosting business, a dedicated server depicts the rental and exclusive usage of a computer that comprises a web server and its related software along with an Internet connection. Usually, the dedicated servers connected to the Internet are present at the web hosting company’s premises only. A business needs dedicated servers for websites that may drive quite significant traffic volume in a day. However, the dedicated servers can also be configured and operated remotely from the client’s end as well. While renting out a dedicated server, the client might need a single system or a cluster of similar operating systems. A rented dedicated server has a static amount of memory, hard disk space along with bandwidth.Types of BackupsNow that we have understood what dedicated servers are let’s understand what are the different types of backups that you can have-RAID BackupsRAID is a simple method that can be introduced for enhancing the data security that is hosted on your dedicated servers. RAID refers to a hardware setup that needs two physical disks. The underlying concept is- that when data is stored on one of the hard drives, it automatically gets replicated on the second hard ...


Read More on Datafloq

Sunday, 26 April 2020

Advantages of Data-driven Marketing Over Traditional Marketing

With the technological advances in big data and how we collect, process, and analyze it, marketing as we know it has changed through the years. Data-driven strategies have pushed the envelope when it comes to predicting customer behavior and adapting approaches accordingly. It allows for the creation of relevant experiences that are tailor-made to address customer demands and expectations.Data-driven marketing is considered the next generation of marketing because it provides insights into customers that were very hard to come by before. While marketing instinct is helpful, it’s just intelligent guessing at best, if not backed by relevant data. A data-driven marketing strategy involves the collection and processing of large data sets to help predict customer behavior in relation to market positioning, brand perception, and new products or feature launches.Why Choose Data-driven Marketing?In today’s always-connected, always-online world, customer expectation has jumped and businesses have ramped up their strategies to meet them. Data-backed metrics will help transform businesses and maximize success both online and offline. Here are a few reasons why you should adopt a data-driven approach in your marketing efforts today.Data helps clarify your target audience.Any type of data that helps you understand your target audience is relevant data. There are ...


Read More on Datafloq

Friday, 24 April 2020

Improving Data Quality for the Insurance Industry – Why and How

How does an insurance company determine who is and who isn’t eligible for a policy? How do they decide on the premium that must be paid? How do they know which claims should be settled and which are fraudulent? The answers to all these questions can be unlocked using data.As we all know, insurance is a customer centric industry and highly dependent on data. It would be fair to call data the foundation for the industry. Just as how a house built on a weak foundation will collapse, an insurance company that uses poor quality data cannot be expected to be successful. Let’s find out how data quality can make or break your insurance business. 4 Ways poor data quality affect the insurance industry?Well, improving data quality is a fundamental challenge for this sector. Some of the ways it can create roadblocks for the insurance industry are:1. It Compromises Customer ExperienceApart from the policies offered, it’s the ease of applying for an insurance policy and the customer experience that keeps customers loyal to a company. If the company uses poor quality data, communication between them and the customer will be flawed. This could be as ...


Read More on Datafloq

Software Defined Perimeter: A Quick Guide of Types and Security Benefits

It is the successive wave of development in the arena of technology that has redefined the world with myriad of marvels. The business world is harnessing the benefits of cloud and IoT security.

Cloud is going to influence and dominate the business world for quite a few years with its wonderful features such as pay-per-use pricing, cheap storage, on-demand resources, and disaster recovery.

With the convenience and promising features of advanced tech, the incidents of security breaches are increasing every single day. An upsetting number of data breaching incidents have been recorded in recent times which jeopardized the efficacy of the cloud technology, the most innovative infrastructure advancement of contemporary times.

To secure the efficiency of the cloud security solution, a new approach has emerged known as Software-Defined Perimeter (SDP).

Definition of Software-Defined Perimeter

Software-Defined Perimeter is an innovative approach to the security of computer systems in any organization that micro-segments network access and enables direct connections between the users and the resources they access.

According to Gartner, "SDP enables organizations to provide people-centric, manageable, secure and agile access to networked systems. It is easier and less costly to deploy than firewalls, VPN concentrators and other bolt-in technologies.”

A Software-Defined Perimeter has three main pillars. They are:

Zero ...


Read More on Datafloq

IEEE Initiative on Ethical Design Is Making Headway

By John P. Desmond, AI Trends Editor

A three-year effort by hundreds of engineers worldwide resulted in the publication in March of 2019 of Ethically Aligned Design (EAD) for Business, a guide for policymakers, engineers, designers, developers and corporations. The effort was headed by the IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems (A/IS), with John C. Havens as Executive Director, who spoke to AI Trends for an Executive Interview. We recently connected to ask how the effort has been going. Here is an update.

EAD First Edition, a 290-page document which Havens refers to as “applied ethics,” has seen some uptake, for example by IBM, which referred to the IEEE effort within their own resource called Everyday Ethics for AI  The IBM document is 26 pages, easy to digest, structured into five areas of focus, each with recommended action steps and an example. The example for Accountability involved an AI team developing applications for a hotel. Among the recommendations was: enable guests to turn the AI off, conduct face-to-face interviews to help develop requirements; and, institute a feedback learning loop.

The OECD (Organization for Economic Cooperation and Development) issued a paper after the release of an earlier version of EAD attesting to the close affinity between the IEEE’s work and the OECD Principles on AI. The OECD cited as shared values “the need for such systems to primarily serve human well-being through inclusive and sustainable growth; to respect human-centered values and fairness; and to be robust, safe and dependable, including through transparency, explainability and accountability.”

John C. Havens, Executive Director, IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems

Based on the success of working to produce a more focused document based on the full EAD, Havens said he is now working with other groups to produce focused guides on EAD for parents, for artists, for the financial industry, as well as for health and for sustainability.

It can be difficult for the IEEE effort not to get buried in the myriad of principles documents surrounding ethical AI development, “maybe 80 of them,” Havens said. Usually the IEEE does get mentioned prominently. Since its release, the EAD has spawned 13 standards working groups within IEEE, and two more will be launched in coming months. “A more pragmatic instantiation of the ideas will come to light,” Havens said.

Another effort is around an effort to enable programmers, engineers, technologists and business managers to better consider how the products and services they create can increase human well-being. (See IEEE P7010 Well-Being Metrics Standard for Autonomous and Intelligent Systems.) Well-being is defined in the draft document by a spectrum of measures, including environmental and educational issues, feelings and mental health.

“The logic is how to have an applied ethics foundation methodology that syncs with how engineers work,” said Havens. “We’re trying to make it easier for them to get the cross-pollinated support they need from non-technical colleagues.” He noted that many of the contributors to the EAD for the IEEE are in academia, so that its principles are likely to be incorporated into college curriculums, where students will be absorbing it.

Asked if he could cite an example of how the IEEE’s ethical AI effort is playing out, Havens mentioned Dr. AJ Moon, an experimental roboticist now on the faculty of McGill University in Montreal, who is on the executive committee for The IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems  that produced the EAD. She is also the founder and director of the Open Roboethics Institute.  That site is publishing research on topics such as whether AI in healthcare has been “ethics-proofed.”

In the area of personal data privacy, the EAD talks about “data agency” and new ways individuals can grant permission for how their data is used. Asked for an example of how this is playing out, Havens mentioned an app called Private Kit: Safe Paths, developed by a team at the MIT Media Lab led by Ramesh Raskar and included developers from Harvard, Facebook and Uber.

This app is now being used to help track the spread of the coronavirus. It works by sharing encrypted location data between phones in a way that does not go through a central authority. In this way, a user can see if they might have come in contact with someone carrying the coronavirus, without knowing who it might be. A person using the app who tests positive, can choose to share location data with health officials, who could then make it public.

Getting the word out is now the challenge for that effort. “The user adoption strategy will leverage network effects,” Raskar stated in a recent interview in Digital Trends. “We built a web tool for health authorities to disseminate privatized trails. We will use this two-sided network effect to first push the health players in focused, localized site, then let users nudge their acquaintances so collectively they have a ‘peace of mind.’”

Before joining IEEE, Havens worked as a consultant for Gliimpse, a company acquired by Apple in 2016, that practiced “data sovereignty” in the way it allowed personal health data to be shared with healthcare providers directly. The company was founded in 2013 by Anil Sethi and Karthik Hariharan. The software essentially allowed an individual to compile a personal Electronic Health Record, to be shared directly to healthcare providers.

“Gliimpse provided a portable version of your health data, which empowers you as a patient,” Havens said. It can also be used to send healthcare information on children to summer camps, for example.

Sethi later left Apple to form Citizen, a consumer health technology company with a mission to give patients control of their health records. He started the company after the loss of his sister Tamira to breast cancer; she made him promise to do something about health care before she died.

The IEEE Ethics effort is also concerned with defining “well-being metrics.” Overall, well-being is defined as the experience of health, happiness and prosperity, according to Psychology Today. Fortunately, lots of related measurable data is provided in the latest version of the World Happiness Report, recently issued by the Sustainable Development Solutions Network, which is supported by a wide range of impressive organizations and individuals, including the Gallup World Poll. “Gallup has the most data on individually oriented well-being,” Havens said.

The UN report conveys the notion of “self-reporting” on life satisfaction on a zero to 10 scale, with zero being pure misery. This notion of providing a Life Satisfaction score is based  on where a person lives in the world and their subjective reporting of their individual experience. The UN report explores, for example, why citizens of Nordic countries have a high life satisfaction. “The point for our work is that well-being needs to be taken into account by engineers or anyone when designing systems,” said Havens. “The UN report provides specific quantitative and qualitative empirical data to help anyone creating AI Systems to do just that.”

In a new related initiative, the IEEE is launching a certification program later in the year that will provide a mark certifying that a product or service has met the criteria for effective accountability, transparency or protection against data bias, as example categories.

Learn more at EAD First Edition.

Baby-On-Board and AI Autonomous Cars

By Lance Eliot, the AI Trends Insider

You’ve undoubtedly seen the famous Baby On Board signs that were a crazed one-hit wonder during the mid-1980s.

When the fad first emerged, it seemed like these ubiquitous yellow-colored signs and their bumper sticker variants were popping up on cars everywhere. The moment that you pulled onto the street from your house, you’d likely see the Baby On Board prominently on the backs of cars streaming down your street, assuming you lived in an area that had young families that were eager to announce their baby occupant.

At first, it seemed that most people genuinely placed such a sign in the rear window of their cars as a means of letting other drivers know that there was a baby or small child inside the moving car.

The notion was to forewarn other drivers to be especially careful when driving near to the car, presumably wanting to make sure that other drivers were supposed to regard the car as “special” since it contained a baby.

I remember that some drivers that didn’t have children or had children that were far beyond the baby age were somewhat disturbed at the emergence of these signs.

Did the sign imply that other drivers were callous around other cars and that by posting a Baby On Board sign they should clean-up their act?

It was kind of a back-handed insult to these other drivers. One interpretation was that you, the person reading the sign, were a really crummy driver and that by announcing the presence of a baby, you were having a finger wagged at you to not be such a careless and witless driver.

There were some drivers that actually then became upset and decided to purposely drive recklessly near such a car, trying to show that other car who’s the boss. The appearance of the Baby On Board sign became a kind of beacon for those affronted drivers. This might seem nutty now, but whenever a widespread fad like that emerges, there are bound to be some that don’t like the fad and profess to make it backfire.

Rumors abounded that the sign was actually intended to alert emergency services or first responders whenever they came upon a car accident scene.

If there was a wreck of cars, presumably the bright yellow sign would stand out and the fire department responding would know to find a baby inside that particular car. Some assumed that there was apparently a high chance that your baby might get overlooked, and maybe you as an adult sized body would be dragged out of a decimated car, but your baby would get left behind.

The original developers and firm that brought the Baby On Board to worldwide attention had mainly in mind the idea of forewarning other drivers to be careful when near to a car with a baby in it. There is scant evidence to suggest that somehow babies weren’t being pulled out of car wrecks.

Nor was there evidence that once the signs became popular that somehow it raised the chances of a baby being discovered that was otherwise going to be overlooked in a demolished car.

For some people, the sign shifted in meaning toward a source of pride that they had a baby, regardless of whether the baby was actually in the car at the time or not.

In other words, many people bought the signs and put them on their cars, doing so as a showcase that hey, I’ve got a baby, and congrats that I have one. You might not have normally been telling the world that you had a baby per se, but this sign made it easy to make such an announcement.

Of course, having the sign on your car when you didn’t actually have your baby in the car was kind of defeating the purpose of the sign. If you were inclined to believe in the theory that the sign would give a crucial clue to first responders to look for a baby in a wrecked car, the sign was now at times going to endanger first responders by having them search in vain for a baby left behind. This would increase the risk for the first responders and quite adversely undermine the matter. It made things potentially worse rather than for the better.

This aspect of leaving the sign in your car window when you didn’t have a baby in the car was yet another source of aggravation for some other drivers.

Those drivers that were perturbed at the prevalence of cars using the sign and yet didn’t have a baby in the car were at times tempted to look inside your car as you drove past. If they did not see evidence that a baby was in your car, these irked drivers would sometimes honk their horn at the car or try to perform untoward maneuvers nearby the offending car and its offending driver in a kind of retaliation. Take down your sign, some of them would exclaim in anger.

Another viewpoint about the baby in the car warnings was that it maybe was good for the driver of that car that had the baby, causing that driver to be more cautious in their driving.

Allow me to explain.

Suppose the driver of a car that had a baby in it was in front of you and suddenly cut you off in traffic. Well, that’s a move that endangers the baby in that car. Some hoped that the people putting the sign onto their car would become more thoughtful drivers as a result of their own realization that they had a baby in their car. In that manner, the sign is really for the driver of the car that has the baby in it, more so than to alert other drivers about the car with the baby in it.

Eventually the Baby On Board signs became a kind of meme, akin to the nature that we have today on social media whenever something catches the fancy of the public at large. Parodies sprung up and it became popular to put a faked version on your own car window. For example, there was the Baby Driving version, the Mother In-Law In Trunk version, and the Baby Carries No Cash version, and so on. Millions of the bona fide versions were sold and some estimates say that many millions more of the parody versions were sold.

Downsides Of Baby-On-Board Signs

Use of the sign on your car was considered questionable in other ways.

For example, some states in the United States got worried that the signs would use up space on your car window and obstruct your view.

It was even outlawed with a ticketed violation in some jurisdictions, prohibiting you from putting one on your car window. Another concern was that the signs were distracting drivers and causing them to focus on reading the sign rather than watching the road and traffic conditions. Since the sign didn’t seem to have much of a greater useful purpose per se, the distraction factor made it overly dangerous in comparison to whatever benefit it might provide.

There are various urban myths and other fascinating tales during the heyday of the Baby On Board signs.

One popular tale was that drug smugglers would at times use the sign, in hopes of fooling the police into assuming that a car that had drugs would most certainly not have drugs, because of course no one would put illegal drugs into a car that had a cute innocent baby in it.

Partially due to the confusion about the sign and the backlash, the fad eventually waned.

Nonetheless, the sign and the saying became an enduring icon. You would be hard-pressed to find anyone that did not know about the Baby On Board signs in the sense that they’ve seen one either in real-life or seen it online someplace. The younger generation that did not grow-up with the signs have often heard about the signs or seen the signs and slogan portrayed in a variety of TV shows, movies, online games, and in a wide variety of other ways.

There are offshoots of the Baby On Board usage, such as wearing such a sign as woven into an apparel item worn by a pregnant woman. Some view this as a clever way to say a baby is on its way. There is a nostalgic side of the market too. You can still get the signs and put them onto your car, which you’ll see from time-to-time on the roads. As usual, not everyone is keen on the use of the sign. There are some that say you are opening yourself to notifying prospective kidnappers or other hoodlums that might be interested in grabbing your baby, or that they can use the sign to strike up a conversation with you, acting as though they know you, and try to scam you in some other manner.

It’s a cruel world out there.

Baby On-Board Aspects

Let’s for the moment consider what it means to truly have a baby on-board of your car.

Putting aside the matter of the famous sign, what kinds of things should people be doing if they actually are carrying a baby in their car. This might be especially instructive for those of you that have yet to drive a car with a baby in it.

Perhaps the most attention and discussions about having a baby inside a car involves the use of a baby seat for the child.

Going all the way back to the 1920s, the early versions of baby car seats were essentially sawed-off high-chairs that had straps and some other restraints on them. The focus was to simply keep the baby from being able to move around in the car. There wasn’t much thought given to the safety of the baby and nor what might happen to the baby when the car got into an accident or performed some radical driving maneuver.

Until the late 1960s, most baby seats were about the same in terms of lack of careful consideration for what a baby seat should do. The auto makers began to provide so-called love seats and guard seats for housing a baby inside a car during the latter part of the 1960s, and then during the 1970s the United States began regulating the safety aspects of baby seats.

At one point, there was almost a baby seats “war” in which different baby seat makers vied to get parents to buy their particular brand and models of baby seats. Do you have love in your heart for your baby? Would you give anything to protect your baby? If so, it seemed that the baby seat makers would shame you into buying the most expensive baby seat they could make. The more the baby seat looked like an astronaut’s seat, it was assumed by many parents that they were doing the right thing by buying such an elaborate contraption.

During this time period, there was research undertaken that indicated having the baby ride in the backseat is much safer than having the baby ride in the front seat. This seems rather intuitive in that you figure that any accident is likely to smash or cause flying debris to appear in the front seat, and so by placing the baby in the back seat you are essentially cocooning them further away from the mayhem.

There was also research that indicated the baby should be placed in a rear-seat facing manner. Thus, not only should the baby be in the back seat, the baby should also be facing toward the back of the car. You can imagine that some parents were dubious about this approach. How could they watch their baby and make sure the baby was okay during a driving journey? Which was more important, having the baby in the presumed proper position for a car crash, which might happen once in a blue moon, or have the baby facing forward such that via glancing back or looking in the rear view mirror the parents could instantly see the status of the baby.

Another factor became how to affix the baby seat into the car.

Sadly, many of the souped-up baby seats were difficult to install into the actual seat of a car, and thus some parents discovered to their dismay that though they spent a fortune on the baby seat, it did little good during a crash because the baby seat itself was not well affixed in the car. This led to a massive campaign to try and explain to parents how to install their baby seats.

To this day, it remains a potential difficulty and concern.

The baby seat makers were able to realize that parents wanted not only a car baby seat but also wanted other kinds of seating for their baby. You might have a stroller that your baby sits in and have a separate car baby seat. If you went for a trip on a plane, you’d need to take your baby stroller and your separate baby seat too. Knowing how to use both of those completely incompatible contraptions and their idiosyncrasies made it more arduous to use them. This led to the all-in-one approach of devising baby seats that worked for cars and for strollers and for other purposes too.

Though the car-related baby seat topic tends to dominate attention about having a baby in a car, I’d like to cover various other elements involved in having a baby in a car as well.

One aspect that I alluded to already involves the desire to keep tabs on the baby. An adult in the car should presumably be making sure that the baby is doing okay. This would usually involve trying to watch the baby and see how the baby is doing. You might also be listening to determine whether the baby is happy or maybe crying. The baby might also wiggle around and be flailing, the movement of which might make noise or might catch your visual attention out of the corner of your eye.

If you are driving a car and it is just you and the baby in the car, this desire to drive well and pay attention to the baby can be challenging.

It would be one thing too if the baby was seated adjacent to you in the front of the car, but the baby being in the backseat makes it even more arduous to keep tabs on the baby. I’ve seen many adults transfixed on their rear-view mirror, trying to watch their baby, and angling the rear-view mirror downward rather than keeping it in position to see the cars behind them.

Sometimes the driver will repeatedly glance over their shoulder, turning their head away from the traffic ahead. This is another dangerous gambit, similar to trying to use the rear-view mirror to watch your baby. For each moment that you think you are doing the right thing by looking at your baby, you are likely increasing the chances of getting into a car accident. It’s a tough balance. Do you not try to keep tabs on your baby and risk the baby somehow having troubles and you don’t realize it, but in doing so you are maybe putting the baby into greater danger due to driver distraction?

Remember earlier about the Baby On Board signs and that some believed it was really supposed to be for purposes of getting the driver with the baby on-board to be safer? Part of the belief was based on the idea that drivers with a baby in their car are going to axiomatically be distracted and therefore be a worse driver. In fact, some of the other drivers that were irked at the drivers with babies in their cars was that those such drivers tend to often make late lane changes or do other acts that suggested they were distracted and only periodically watching the road.

Which is more important, your catching the drool coming from your babies’ mouth, or making a proper lane change when going at 65 miles per hour freeway speeds? Trying to be a one-person band when driving a car is often quite arduous when you have a baby in the car. There you are, worrying about the baby, worrying about the traffic, and who knows what else might be on your mind. It’s a lot to deal with as a driver.

If you have another adult in the car with you, hopefully the other adult will assume the duties of keeping tabs on the baby. That’s the theory, though of course it can sometimes merely splinter the attention of the driver even further. The driver might be watching the other adult to make sure the adult is properly keeping tabs on the baby, and meanwhile the driver is trying to still on their own keep tabs on the baby. You’ve now got the driver contending with two living creatures at the same time. It’s not always the case that this will lessen the distractions of the driver, but at least it provides the possibility.

Nuances Of Babies On-Board

Now that I’ve covered the aspects about the baby seat and the difficulties of keeping tabs on your baby while you are driving a car, let’s consider some other elements too.

Suppose you put the baby into your proper baby seat and the baby seat is correctly secured in your car.

Good so far.

You opt to go on a leisurely drive along the coastline. It’s a gorgeous sunny day. You drive along the coast highway and admire the ocean and the sunshine. Unfortunately, you failed to consider the sun exposure that the baby is getting in the backseat of the car. The baby is likely not able to realize the dangers of sun exposure and nor alert you to the aspect they are getting sunburned (as you know, adults get sunburned all the time, not realizing it is happening until long after it occurs).

This brings up the importance of making use of sunscreens, either physical ones on the car windows or attached to the baby seat, or some kind of protection for the baby from the sun rays. That’s another element to be considered when you have a baby in your car.

I’ll mention some additional and quite serious and disconcerting dangers (brace yourself or skip ahead to the next section).

Suppose you put the baby into the baby seat and have put the baby seat next to the rear window. A baby might be able to open the window and endanger themselves. Maybe even open the car door. This is the reason that many cars now have childproof locks on the rear windows and doors. This is yet another element to keep in mind about having a baby in the car.

Lots of other dangers are possible.

You might accidentally close a window on a baby’s appendage, or likewise close a door on them.

You might leave the baby in the car and the baby could get injured or die from heat stroke.

While driving the car, if you take a turn or curve in a harsh manner, it could toss the baby around, in spite of the baby seat protection. The baby’s limbs might not be conducive to sudden braking or sudden accelerations. There are some doting parents that perhaps become overly protective as drivers and they go exceedingly slow and take turns agonizingly sluggishly, which though you could say is the right kind of spirit, can actually increase the chances of getting into a car accident. Becoming a hazard on the roadway due to overly cautious driving can be a downfall for a loving parent.

The last point on this topic for now is that there is still that question about what happens to the baby when an emergency occurs.

As mentioned earlier, the Baby On Board sign was not especially adopted to deal with making sure that first responders would know to save a baby in a wrecked car. From a slightly different viewpoint, there is the matter of how a parent can best extricate their baby from a car if there is an emergency. Do you try to remove the baby from the car seat and then remove the baby from the car? Or, would it be more prudent to remove the entire baby seat with the baby in it? You might have only split seconds to decide what to do and therefore should have considered beforehand what you will do.

AI Autonomous Cars And Baby On-Board

What does this have to do with AI self-driving driverless autonomous cars?

At the Cybernetic AI Self-Driving Car Institute, we are developing AI software for self-driving cars. One important “edge” problem involves having the AI be of assistance when you have a baby inside the self-driving car.

Allow me to elaborate.

I’d like to first clarify and introduce the notion that there are varying levels of AI self-driving cars. The topmost level is considered Level 5. A Level 5 self-driving car is one that is being driven by the AI and there is no human driver involved. For the design of Level 5 self-driving cars, the automakers are even removing the gas pedal, brake pedal, and steering wheel, since those are contraptions used by human drivers. The Level 5 self-driving car is not being driven by a human and nor is there an expectation that a human driver will be present in the self-driving car. It’s all on the shoulders of the AI to drive the car.

For self-driving cars less than a Level 5 or Level 4, there must be a human driver present in the car. The human driver is currently considered the responsible party for the acts of the car. The AI and the human driver are co-sharing the driving task. In spite of this co-sharing, the human is supposed to remain fully immersed into the driving task and be ready at all times to perform the driving task. I’ve repeatedly warned about the dangers of this co-sharing arrangement and predicted it will produce many untoward results.

For my overall framework about AI self-driving cars, see my article: https://aitrends.com/selfdrivingcars/framework-ai-self-driving-driverless-cars-big-picture/

For the levels of self-driving cars, see my article: https://aitrends.com/selfdrivingcars/richter-scale-levels-self-driving-cars/

For why AI Level 5 self-driving cars are like a moonshot, see my article: https://aitrends.com/selfdrivingcars/self-driving-car-mother-ai-projects-moonshot/

For the dangers of co-sharing the driving task, see my article: https://aitrends.com/selfdrivingcars/human-back-up-drivers-for-ai-self-driving-cars/

Let’s focus herein on the true Level 5 self-driving car. Much of the comments apply to the less than Level 5 self-driving cars too, but the fully autonomous AI self-driving car will receive the most attention in this discussion.

Here’s the usual steps involved in the AI driving task:

  • Sensor data collection and interpretation
  • Sensor fusion
  • Virtual world model updating
  • AI action planning
  • Car controls command issuance

Another key aspect of AI self-driving cars is that they will be driving on our roadways in the midst of human driven cars too. There are some pundits of AI self-driving cars that continually refer to a utopian world in which there are only AI self-driving cars on the public roads. Currently there are about 250+ million conventional cars in the United States alone, and those cars are not going to magically disappear or become true Level 5 AI self-driving cars overnight.

Indeed, the use of human driven cars will last for many years, likely many decades, and the advent of AI self-driving cars will occur while there are still human driven cars on the roads. This is a crucial point since this means that the AI of self-driving cars needs to be able to contend with not just other AI self-driving cars, but also contend with human driven cars. It is easy to envision a simplistic and rather unrealistic world in which all AI self-driving cars are politely interacting with each other and being civil about roadway interactions. That’s not what is going to be happening for the foreseeable future. AI self-driving cars and human driven cars will need to be able to cope with each other.

For my article about the grand convergence that has led us to this moment in time, see: https://aitrends.com/selfdrivingcars/grand-convergence-explains-rise-self-driving-cars/

See my article about the ethical dilemmas facing AI self-driving cars: https://aitrends.com/selfdrivingcars/ethically-ambiguous-self-driving-cars/

For potential regulations about AI self-driving cars, see my article: https://aitrends.com/selfdrivingcars/assessing-federal-regulations-self-driving-cars-house-bill-passed/

For my predictions about AI self-driving cars for the 2020s, 2030s, and 2040s, see my article: https://aitrends.com/selfdrivingcars/gen-z-and-the-fate-of-ai-self-driving-cars/

Returning to the topic of having a Baby On Board, let’s consider how the AI can be of assistance if you do indeed have a baby inside a self-driving car.

For an AI self-driving car that is less than a true Level 5 or Level 4, there should be a human driver in the car, and in that case the AI could potentially assist the driver in ways other than driving the car per se, such as monitoring the baby and relaying status of the baby to the human driver.

The AI could use the interior-facing cameras of the self-driving car to do facial recognition about the baby.

Does the baby look okay, or is the baby turning blue because of swallowing something that might be blocking their airway passage?

The emotional state of the baby can be potentially interpreted via the facial expressions and movement of the baby while in the baby seat. The self-driving car also will have an audio microphone inside the car that can be used to listen for sounds, including the baby crying or the baby cooing.

It is likely that the baby seat will have its own Internet of Things (IoT) devices such as detecting the heart rate of the baby and other vital signs, of which this data can be conveyed to the AI of the self-driving car. There is also a likelihood that the AI can ascertain whether the baby seat is properly secured within the self-driving car. This would be based on sensors within the seats of the self-driving car and also in combination with the camera images showing whether the baby seat is slipping around or staying in place.

The human driver would presumably be able to focus on the driving tasks expected of the human and would feel reassured that the AI is monitoring the status of their baby inside the car. The AI would be able to provide a verbal indication to the human driver about the status of the infant. This could be an interactive dialogue based on the Natural Language Processing (NLP) capabilities of the AI system that are incorporated to engage the human in discussions about the driving task.

Beyond the simpler status aspects, the AI if specially developed with capabilities for assisting in baby monitoring would be able to detect some of the more potentially dangerous aspects that can befall the baby.

For example, if the baby is seated close to a car window, the AI via the visual image processing could detect if the baby tries to go outside of the window or tries to get the car door open. The AI could detect when the human driver opts to leave the car and hopefully therefore ascertain if the baby is being left behind in the car, reducing the chances of hot car deaths when an infant is inadvertently not removed from a car when needed. The AI might be able to detect sun exposure that the baby is getting and warn the human driver or possibly adjust the windows automatically to block the sun’s rays. And so on.

For more about the Internet of Things (IoT) see my article: https://www.aitrends.com/selfdrivingcars/internet-of-things-iot-and-ai-self-driving-cars/

For the use of NLP, see my article: https://www.aitrends.com/selfdrivingcars/car-voice-commands-nlp-self-driving-cars/

For the use of AI self-driving cars for family trips, see my article: https://www.aitrends.com/selfdrivingcars/family-road-trip-and-ai-self-driving-cars/

For the non-stop use of AI self-driving cars, see my article: https://www.aitrends.com/selfdrivingcars/non-stop-ai-self-driving-cars-truths-and-consequences/

For car crashes and the AI systems, see my article: https://www.aitrends.com/selfdrivingcars/accidents-contagion-and-ai-self-driving-cars/

Added Aspects Of AI Aiding The Baby-On-Board

When an AI self-driving car gets involved in a car crash or other disabling action, if the AI system is still functioning it could act as a kind of Baby On Board alert capability. This could involve the AI system getting the car to perhaps honk its horn as a sign that there is a baby in the self-driving car (though this is obviously a dubious means and could be misunderstood). The self-driving car might have external e-billboards that the AI could use to display a message for first responders that indicates a baby is inside the car (assuming the e-billboards are still functioning).

The AI could use V2V (vehicle-to-vehicle) electronic communication to potentially send out a message indicating that a baby is inside the car, which might then get picked-up by responding vehicles and relayed to first responders. Likewise, the AI might use V2I (vehicle-to-infrastructure) electronic communications to inform a nearby element of roadway infrastructure and of which the first responders might also be in contact with the roadway edge computing devices.

Though I’ve emphasized these various communication means to inform others about a baby inside the car, I’d like to also mention that this same kind of messaging could be used to indicate more aspects beyond the notion of a baby being on-board the car. The AI could indicate how many occupants there are in the self-driving car. The AI might be able to ascertain the general medical status of the occupants, such as whether they are still breathing or not and how injured they are. The AI could provide the status of the car itself such as whether it is still running or if it is crumpled up.

All such information would be handy for emergency responders as they seek to get to the scene of the car accident.

For more about edge computing, see my article: https://www.aitrends.com/selfdrivingcars/edge-computing-ai-self-driving-cars/

For safety aspects, see my article: https://www.aitrends.com/selfdrivingcars/safety-and-ai-self-driving-cars-world-safety-summit-on-autonomous-tech/

For my article about the dangers of getting out of a moving car, see: https://www.aitrends.com/selfdrivingcars/shiggy-challenge-and-dangers-of-an-in-motion-ai-self-driving-car/

For the analysis of self-driving car post-mortems, see my article: https://www.aitrends.com/selfdrivingcars/pre-mortem-analysis-for-ai-self-driving-cars/

So far, I’ve focused on the AI acting as a kind of assistant to the human driver that is present in the self-driving car and trying to be a nanny to watch over the baby.

Would this help ensure that the human driver remains more attentive to the driving task?

In other words, if the human driver knows that the baby is being closely monitored and the AI is informing the driver about the status of the baby, perhaps the driver would no longer feel compelled to turn their head to look at the baby seated in the back seat or try to watch the baby via a rear-view mirror.

You could counter-argue that the AI attention might spur the human driver toward being less attentive to the driving task. For some drivers, perhaps they might normally glance at the baby on a periodic basis when unaided by the AI. If the AI is continually giving status updates, it could spark the human driver to become more aware of the baby and therefore divert their attention toward the baby more so. Of course, that’s the opposite of what the AI monitoring is supposed to be achieving. A well devised AI monitoring system would presumably inspire confidence in the driver that the baby is being monitored sufficiently.

There is also the factor that the human driver is also likely being monitored by the AI. Whenever the human driver turns their head, the internal-facing camera would be able to detect this head turning. The AI would be able to gently caution the driver about diverting their focus away from the driving task. In that sense, even if the human driver is somewhat inadvertently sparked to look at the baby, the AI can help the human driver to realize this is occurring.

Babies And Other Passengers

One question that is a bit thorny involves what to do if the AI detects that something untoward is happening with the baby, whether real or imagined.

I remember when my children were babies and at one point, I had one of them in their baby seat in the backseat, and a passenger in the car turned to look and suddenly exclaimed “Oh my gosh!” as though something horrible had just happened.

This greatly startled me, and I assumed that somehow my offspring had gotten injured or was choking or something really bad was taking place.

Instead, it was simply that a toy had been ripped apart and the fluffy pieces were all floating around the backseat of the car.

The passenger had reacted a bit over-the-top to this.

There was no immediate danger involved. I didn’t know that status at the instant of the exclamation and so I reacted rapidly by immediately pulling over to the side of the road. This was a dramatic car maneuver and not one that I would normally have made. After realizing that this was not a true emergency, I cautioned the passenger that in the future they ought to be more careful as to how they react to such circumstances and be thoughtful of the impact it could have on the car driver.

How should the AI convey a potential urgency to the human driver if the AI detects something is amiss about the baby?

I doubt that we want the AI to make any loud exclamations or otherwise startle the human driver. The AI would need to balance between informing the human driver and not otherwise causing the human driver to make any sudden and untoward reactive actions.

Let’s next consider what the AI can do about a baby being inside a self-driving car when the self-driving car is at a true Level 5.

Since the true Level 5 self-driving car does not need a human driver, this implies one of several possibilities about the baby being inside the self-driving car.

First, it could be that there is an adult in the self-driving car and essentially an occupant with the baby.

In that case, the adult would hopefully be monitoring the baby. The AI could still be monitoring the baby, perhaps as a double-check or as further assistance to the adult. If the adult is perhaps weak in their faculties, maybe an elderly grandparent that is not so able to tend to the baby, the AI might serve as an adjunct to the adult.

Second, it could be that there is not an adult in the self-driving car and only a minor that accompanies the baby.

This is an easy scenario to imagine. Suppose you decide to have your AI self-driving car drop-off your two children at grandma’s house. You are too busy to go along. You put your 6-year-old daughter and your baby boy into the self-driving car, and you command the AI to take them to grandma’s.

You are making an assumption that your 6-year-old daughter can take care of the baby during the driving journey. Though you might believe that to be the case, in most states you are likely violating a provision about making sure that an adult is accompanying your baby. A minor is not considered the equivalent of having an adult present.

I’m sure that people will be tempted to assume that with the AI monitoring the baby and their daughter, and with the likelihood too of the remote parent being able to electronically communicate with the AI self-driving car, such as watching the camera feeds, this will be sufficient to then allow their unaccompanied children to be driving around in the AI self-driving car. Parents that are pressed for time will consider this handy as a means of transporting their children for them.

I’d say that we are heading toward a societal, ethical, and regulatory matter that will require discussion and debate.

Right now, if you do ridesharing for your children, they nonetheless still have an adult in the car, namely the ridesharing driver. There is in theory no means for you to currently put your underage children into a car and have that car go anyplace without at least one adult present, namely the human driver.

For my article about the rise of ridesharing and AI self-driving cars, see: https://www.aitrends.com/selfdrivingcars/ridesharing-services-and-ai-self-driving-cars-notably-uber-in-or-uber-out/

For my article about ethical review boards, see: https://www.aitrends.com/selfdrivingcars/ethics-review-boards-and-ai-self-driving-cars/

For my article about Gen Z and AI self-driving cars, see: https://www.aitrends.com/selfdrivingcars/gen-z-and-the-fate-of-ai-self-driving-cars/

For regulations about AI self-driving cars, see my article: https://www.aitrends.com/selfdrivingcars/assessing-federal-regulations-self-driving-cars-house-bill-passed/

Babies On-Board And No Other Humans On-Board

We can make the scenario of having underage occupants even more extreme.

Suppose you put your baby into the baby seat of the AI self-driving car and then command the AI to take your baby over to the house of a babysitter that you use.

During the driving journey, the baby is unaccompanied.

There is no other human inside the self-driving car.

Should we be comfortable with the idea that if the AI is monitoring the baby and suppose that the parent has remote access, we are okay with the baby being in the car by itself?

It seems hard to imagine that we would as a society accept this idea. If the baby suddenly has a severe problem, there is no immediate recovery possible since there is not another human inside the self-driving car.

You might try to claim that the AI self-driving car could try to seek help if the baby is having troubles. Maybe the AI dials 911. Maybe the AI sends out an emergency beacon via V2V and seeks assistance from other nearby cars and their potential human adult occupants. Perhaps the AI drives the self-driving car to the nearest hospital. Yes, these are all possibilities, but they seem rather second-best, at best, in terms of caring for the baby.

We’ll have to wait and see what we opt to do as a society.

Let’s pretend that the practice of having your baby in a true Level 5 AI self-driving car and alone as a human occupant gets outlawed.

We all know that practices that are outlawed are not necessarily ergo no longer undertaken. A parent might opt to normally not put the baby in the AI self-driving car by itself, but perhaps they decide to break the rule, just this once, and do so because they are pressed to do something else and believe they have a good reason to violate this law.

What then?

Well, we could guess that the AI would likely be able to ascertain that the baby is alone in the AI self-driving car. If that’s the case, should the AI then refuse to proceed? Perhaps we have the auto makers and tech firms place a special stop-mode that the AI won’t allow the self-driving car to get underway if there is a baby and no accompanying other human (this has its own challenges too, such as whether the other human is a minor versus an adult, etc.).

Or, suppose the AI self-driving car gets underway, somehow then realizes there is an unaccompanied baby in the car, should it report this aspect to the authorities? Perhaps it calls the police. The AI could drive the self-driving car to the nearest police station or rendezvous with a police car. I realize this seems far-fetched and hard to contemplate, but these scenarios are bound to happen.

If we eventually have hundreds of millions of AI self-driving cars on our roadways, all kinds of things are going to occur in terms of how people decide to make use of an AI self-driving car.

Suppose too that you are a ridesharing firm and you are letting people use your Level 5 or Level 4 AI self-driving cars.

A rider puts a baby into your ridesharing car, tells the AI to go to some destination, and slips out of the self-driving car. During the driving journey, something happens to the baby and it gets injured. Who is responsible for this? Since you provided the ridesharing car, presumably you have some culpability in whatever happens to the unaccompanied baby.

I’ve predicted that we might see a new kind of job role in society, namely the role of being a kind of AI self-driving car “nanny” or caregiver. The person would be hired to ride in an AI self-driving car and be there to accompany minors. They might also be there to aid someone that is elderly and not of full faculties. They might be there to assist in riders getting into and out of the AI self-driving car. Note that the person does not need to know how to drive a car (because the AI is doing the driving), which would reduce the barrier to entry for these kinds of positions. Etc.

For a new job consisting of being a kind of self-driving car nanny, see my article: https://www.aitrends.com/selfdrivingcars/future-jobs-and-ai-self-driving-cars/

For more about responsibility and AI self-driving cars, see my article: https://www.aitrends.com/selfdrivingcars/responsibility-and-ai-self-driving-cars/

For socio-behavioral aspects, see my article: https://www.aitrends.com/features/socio-behavioral-computing-for-ai-self-driving-cars/

For the boundaries of AI self-driving cars, see my article: https://www.aitrends.com/ai-insider/ai-boundaries-and-self-driving-cars-the-driving-controls-debate/

For the robojacking aspects, see my article: https://www.aitrends.com/features/robojacking-self-driving-cars-prevention-better-ai/

AI Driving Efforts And Baby-On-Board

As a final thought for now, let’s assume that there is a baby inside an AI self-driving car, and the baby might or might not be accompanied by another human (as I say, this is yet to be decided by society).

Should the AI self-driving car drive any differently?

Some would assert that the AI should drive the self-driving car in a fully legal and cautious manner, regardless of who or what might be inside the AI self-driving car. I think this is a bit of an over-simplification of the matter. There are degrees of driving that can range from being overly cautious to overly carefree. It is conceivable that the AI can devise a smoother ride for situations such as having a baby inside the AI self-driving car.

When my children were babies, I would definitely be more delicate when I saw a pothole up ahead or a dip in the road. If they had fallen asleep, I would try to avoid any radical turns or fast maneuvers. All of those driving aspects were perfectly legal and none of them were illegal. There is a wide range of discretion in how you drive a car, within the bounds of driving legally.

One aspect too will be the possibility of trying to avoid car sickness for your baby. Adults can get car sick. Babies can also get car sick. I would suggest that a baby is maybe even more generally prone to potential car sickness. The manner of how you drive your car can contribute toward car sickness. In that sense, the AI could adjust its driving approach to try and reduce the chances of car sickness ensuing for a baby, if there is a baby inside the self-driving car.

For my article about car sickness, see: https://www.aitrends.com/selfdrivingcars/kinetosis-anti-motion-sickness-ai-self-driving-cars/

For the illegal driving of an AI self-driving car, see my article: https://www.aitrends.com/selfdrivingcars/illegal-driving-self-driving-cars/

For the human foibles of driving, see my article: https://www.aitrends.com/selfdrivingcars/ten-human-driving-foibles-self-driving-car-deep-learning-counter-tactics/

For the nature of AI driving styles, see my article: https://www.aitrends.com/selfdrivingcars/driving-styles-and-ai-self-driving-cars/

Conclusion

The famous or now somewhat infamous Baby On Board.

This kind of signage is a reminder that for AI self-driving cars, we need to consider the “special case” of what should be done when a baby is inside a self-driving car.

We cannot ignore the matter.

One of the more vexing issues will be whether a baby ought to be riding alone while inside a true AI self-driving car.

The initial reaction would be that the baby should definitely not be alone, but this is something as a society that we have yet to fully address.

Would you want your AI self-driving car to announce that you do have a baby on-board of your self-driving car?

We might see a resurgence of the fad. Via external e-billboards of the self-driving car you might announce it. You might have the V2V let other cars nearby know. Are you doing so for safety purposes or for the desire to brag or for what purpose? Or both?

Well, taking a somewhat lighter perspective and ending this piece on a softer note, we might have a variant of these kinds of signs, one that says AI On-Board.

That’s indeed something we ought to know about.

Copyright 2020 Dr. Lance Eliot

This content is originally posted on AI Trends.

[Ed. Note: For reader’s interested in Dr. Eliot’s ongoing business analyses about the advent of self-driving cars, see his online Forbes column: https://forbes.com/sites/lanceeliot/]