Thursday, 23 April 2020

Glow 0.3.0 Introduces New Large-Scale Genomic Analysis Features

In October of last year, Databricks and the Regeneron Genetics Center® partnered together to introduce Project Glow, an open-source analysis tool aimed at empowering genetics researchers to work on genomics projects at the scale of millions of samples. Since we introduced Glow, we have been busy at work adding new high-quality algorithms, improving performance, and making Glow’s APIs easier to use. Glow 0.3.0 was released on February 21, 2020 and improves Glow’s power and ease of use in performing large-scale, high-throughput genomic analysis. In this blog, we highlight features and improvements introduced in the 0.3.0 release.

Python and Scala APIs for Glow SQL functions

In this release, native Python and Scala APIs were introduced for all Glow SQL functions, similar to what is available for Spark SQL functions. In addition to improved simplicity, this provides enhanced compile-time safety. The SQL functions and their Python and Scala clients are generated from the same source so any new functionality in the future will always appear in all three languages. Please refer to Glow PySpark Functions for more information on Python APIs for these functions. A code example showing Python and Scala APIs for the function normalize_variant is presented at the end of the next section.

Improved variant normalization

The variant normalizer received a major performance improvement in this release. It still behaves like bcftools norm and vt normalize, but is about 2.5x faster and has a more flexible API. Moreover, the new normalizer is implemented as a function in addition to a transformer.

normalize_variants transformer: The improved transformer preserves the columns of the input dataframe, adds the normalization status to the dataframe, and has the option of adding the normalization results (including the normalized coordinates and alleles) to the dataframe as a new column. To start, we use the following command to read the original_variants_df dataframe. Figure 1 shows the variants in this dataframe.


    original_variants_df = spark.read \
        .format("vcf") \
        .option("includeSampleIds", False) \
        .load("/databricks-datasets/genomics/call-sets")

Glow SQL Dataframe with original variants prior to applying the improved variant normalization provided by Glow 3.0.
Figure 1: The variant dataframe original_variants_df

The improved normalizer transformer can be applied on this dataframe using the following command. This uses the transformer syntax used by the previous version of the normalizer:


    import glow
    normalized_variants_df = glow.transform("normalize_variants", \
        original_variants_df, \
    reference_genome_path="/mnt/dbnucleus/dbgenomics/grch38/data/GRCh38_full_analysis_set_plus_decoy_hla.fa" \
    )

Example dataframe, demonstrating the improved variant normalization proved by Glow 3.0, the latest release of the joint open source genomic analysis project.
Figure 2: The normalized dataframe normalized_variants_df

Figure 2 shows the dataframe generated by the improved normalizer. The start, end, referenceAllele, and alternateAlleles fields are updated with the normalized values and a normalizationStatus column is added to the dataframe. This column contains a changed subfield that indicates whether normalization changed the variant, and an errorMessage subfield containing the error message, if an error occurred.

The newly introduced replace_columns option can be used to add the normalization results as a new column to the dataframe instead of replacing the original start, end, referenceAllele, and alternateAlleles fields:


    import glow
    normalized_variants_df = glow.transform("normalize_variants",\
        original_variants_df, \
        replace_columns="False", \
    reference_genome_path="/mnt/dbnucleus/dbgenomics/grch38/data/GRCh38_full_analysis_set_plus_decoy_hla.fa" \
    )

Example Glow SQL normalized dataframe, demonstrating Glow 3.0’s capability to add an additional column with normalization results.
Figure 3: The normalized dataframe normalized_noreplace_variants_df with normalization results added as a new column

Figure 3 shows the resulting dataframe. A normalizationResults column is added to the dataframe. This column contains the normalization status, along with normalized start, end, referenceAllele, and alternateAlleles subfields.

Since the multiallelic variant splitter is implemented as a separate transformer in this release, the mode option of the normalize_variants transformer is deprecated. Refer to the Variant Normalization documentation for more details on the normalize_variants transformer.

normalize_variant function: As mentioned above, this release introduces the normalize_variant SQL expression:


    from pyspark.sql.functions import expr
    function_normalized_variants_df = original_variants_df.withColumn( \
        "normalizationResult", \
        expr("normalize_variant(contigName, start, end, referenceAllele, alternateAlleles, '/mnt/dbnucleus/dbgenomics/grch38/data/GRCh38_full_analysis_set_plus_decoy_hla.fa')") \
    )
As discussed in the previous section, this SQL expression function has Python and Scala APIs as well. Therefore, we can rewrite the previous code example as follows:

    from glow.functions import normalize_variant
    function_normalized_variants_df = original_variants_df.withColumn( \
        "normalizationResult", \
        normalize_variant( \
            "contigName", \
            "start", \
            "end", \
            "referenceAllele", \
            "alternateAlleles", \
        "/mnt/dbnucleus/dbgenomics/grch38/data/GRCh38_full_analysis_set_plus_decoy_hla.fa" \
        ) \
    )
This example can also be easily ported to Scala:

    import io.projectglow.functions.normalize_variant
    import org.apache.spark.sql.functions.col
    val function_normalized_variants_df = original_variants_df.withColumn(
        "normalizationResult",
        normalize_variant(
            col("contigName"),
            col("start"),
            col("end"),
            col("referenceAllele"),
            col("alternateAlleles"),
    "/mnt/dbnucleus/dbgenomics/grch38/data/GRCh38_full_analysis_set_plus_decoy_hla.fa"
        )
    )

The result of any of the above commands will be the same as Figure 3.

A new transformer for splitting multiallelic variants

This release also introduced a new dataframe transformer called split_multiallelics. This transformer splits multiallelic variants into biallelic variants, and behaves similarly to vt decompose with -s option. This behavior is more powerful than the behavior of the previous splitter, which behaved like GATK’s LeftAlignAndTrimVariants with –split-multi-allelics. In particular, the array-type INFO and genotype fields with elements corresponding to reference and alternate alleles are split into biallelic rows (see -s option of vt decompose). So are the array-type genotype fields with elements sorted in colex order of genotype calls, e.g., the GL, PL, and GP fields in the VCF format. Moreover, an OLD_MULTIALLELIC INFO field is added to the dataframe to store the original multiallelic form of the split variants.

The following is an example of using the split_multiallelic transformer on the original_variants_df. Figure 4 contains the result of this transformation.


    import glow
    split_variants_df = glow.transform("split_multiallelics", original_variants_df)
Example Glow SQL dataframe, demonstrating Glow 3.0’s new dataframe transformer called split_multiallelics, which splits multiallelic variants into biallelic variants.
Figure 4: The split dataframe split_variants_df

Please note that the new splitter is implemented as a separate transformer from the normalize_variants transformer. Previously, splitting could only be done as one of the operation modes of the normalize_variants transformer using the now-deprecated mode option. Please refer to the documentation of the split_multiallelics transformer for complete details on the behavior of this new transformer.

Parsing of Annotation Fields

The VCF reader and pipe transformer now parse variant annotations from tools such as SnpEff and VEP. This flattens the ANN and CSQ INFO fields, which simplifies and accelerates queries on annotations. Figure 5 shows the output of the code below, which queries the annotated consequences in a VCF annotated using the LOFTEE VEP plugin.


    from pyspark.sql.functions import expr
    variants_df = spark.read\
        .format("vcf")\
        .load("dbfs:/databricks-datasets/genomics/vcfs/loftee.vcf")
    annotated_variants_df = original_variants_df.withColumn( \
        "Exploded_INFO_CSQ", \
        expr("explode(INFO_CSQ)") \
    ) \
    .selectExpr("contigName", \
        "start", \
        "end", \
        "referenceAllele", \
        "alternateAlleles", \
        "expand_struct(Exploded_INFO_CSQ)", \
        "genotypes" \
    )
Example Glow SQL dataframe, demonstrating Glow 3.0’s ability to parse variant annotations from tools such as SnpEff and VEP.
Figure 5: The annotated dataframe annotated_variants_df with expanded subfields of the exploded INFO_CSQ

Other Data Analysis Improvements

Glow 0.3.0 also includes optimized implementations of the linear and logistic regression functions, resulting in ~50% performance improvements. See the documentation at Linear regression and Logistic regression.

Furthermore, the new release supports Scala 2.12 in addition to Scala 2.11. The Maven artifacts for both Scala versions are available on Maven Central.

Try Glow 3.0!

Glow 0.3 is installed in the Databricks Genomics Runtime (Azure | AWS) and is optimized for improved performance when using cloud computing to analyze large genomics datasets. Learn more about our genomics solutions and how we’re helping to further human and agricultural genome research and enable advances like population-scale next-generation sequencing in the Databricks Unified Analytics Platform for Genomics and try out a preview today.

--

Try Databricks for free. Get started today.

The post Glow 0.3.0 Introduces New Large-Scale Genomic Analysis Features appeared first on Databricks.

How Artificial Intelligence Is Poised To Transform Contact Center Customer Experience

Running a business is all about winning customer experiences, which is best done by being available to them anytime they want to connect. In the era of instant gratification, customers want immediate response to their queries and resolution for their issues. You cannot expect them to wait for getting an answer to an email or queue up with a ticketing solution. They want response and they want it now. No wonder, contact centers have become an essential touchpoint for all businesses that want to win on the availability front. Over the years, their value has grown immensely as they substitute the in-person help that was historically provided at a typical in-store experience. Understandably, businesses are taking all the measures needed to improvise contact centers so that they can stay competitive. The adoption of technologies like cloud and AI are just a few steps in this direction.Let us learn more about the evolution of contact centers before understanding more about the role of Artificial Intelligence in taking them to the next level.Evolution
of Contact Center solutionsContact centers have come a long way since their inception and technology has always been vital to them. During the initial days (when call centers preceded contact ...


Read More on Datafloq

How a Fresh Approach to Safety Stock Analysis Can Optimize Inventory

Refer to the accompanying notebook for more details.

A manufacturer is working on an order for a customer only to find that the delivery of a critical part is delayed by a supplier. A retailer experiences a spike in demand for beer thanks to an unforeseen reason, and they lose sales because of their lack of supply. Customers have a negative experience because of your inability to meet demand. These companies lose immediate revenue and your reputation is damaged. Does this sound familiar?

In an ideal world, demand for goods would be easily predictable. In practice, even the best forecasts are impacted by unexpected events. Disruptions happen due to raw material supply, freight and logistics, manufacturing breakdowns, unexpected demand and more. Retailers, distributors, manufacturers and suppliers all must wrestle with these challenges to ensure they are able to reliably meet their customers’ needs while also not carrying excessive inventory. This is where an improved method of safety stock analysis can help your business.

Organizations constantly work on allocating resources where they are needed to meet anticipated demand. The immediate focus is often in improving the accuracy of their forecasts. To achieve this goal, organizations are investing in scalable platforms, in-house expertise, sophisticated new models.

Even the best forecasts do not perfectly predict the future, and sudden shifts in demand can leave shelves bare. This was highlighted in early 2020 when concerns about the virus that causes COVID-19 led to widespread toilet paper stockouts. As Craig Boyan, the president of H-E-B commented, “We sold in two weeks what we normally sell in two months.”

Scaling up production is not a simple solution to the problem. Georgia-Pacific, a leading manufacturer of toilet paper, estimated that the average American household would consume 40% more toilet paper as people stayed home during the pandemic. In response, the company was able to boost production by 20% across its 14 facilities configured for the production of toilet paper. Most mills already run operations 24 hours a day, seven days a week with fixed capacity, so any further increase in production would require an expansion in capacity enabled through the purchase of additional equipment or the building of new plants.

This bump in production output can have upstream consequences. Suppliers may struggle to provide the resources required by newly scaled and expanded manufacturing capacity. Toilet paper is a simple product, but its production depends on pulp shipped from forested regions of the U.S., Canada, Scandinavia and Russia as well as more locally sourced recycled paper fiber. It takes time for suppliers to harvest, process and ship the materials needed by manufacturers once initial reserves are exhausted.

A supply chain concept called the bullwhip effect underpins all this uncertainty. Distorted information throughout the supply chain can cause large inefficiencies in inventory, increased freight and logistics costs, inaccurate capacity planning and more. Manufacturers or retailers eager to return stocks to normal may trigger their suppliers to ramp production which in turn triggers upstream suppliers to ramp theirs. If not carefully managed, retailers and suppliers may find themselves with excess inventory and production capacity when demand returns to normal or even encounters a slight dip below normal as consumers work through a backlog of their own personal inventories. Careful consideration of the dynamics of demand along with scrutiny of the uncertainty around the demand we forecast is needed to mitigate this bullwhip effect.

Managing Uncertainty with Safety Stock Analysis

The kinds of shifts in consumer demand surrounding the COVID-19 pandemic are hard to predict, but they highlight an extreme example of the concept of uncertainty that every organization managing a supply chain must address. Even in periods of relatively normal consumer activity, demand for products and services varies and must be considered and actively managed against.

Predicted sales as a mean value of actual demand

Modern demand forecasting tools predict a mean value for demand, taking into consideration the effects of weekly and annual seasonality, long-term trends, holidays and events, and external influencers such as weather, promotions, the economy, and additional factors. They produce a singular value for forecasted demand that can be misleading, as half the time we expect to see demand below this value and the other half we expect to see demand above it.

The mean forecasted value is important to understand, but just as critical is an understanding of the uncertainty on either side of it. We can think of this uncertainty as providing a range of potential demand values, each of which has a quantifiable probability of being encountered. And by thinking of our forecasts this way, we can begin to have a conversation about what parts of this range we should attempt to address.

Statistically speaking, the full range of potential demand is infinite and, therefore, never 100% fully addressable. But long before we need to engage in any kind of theoretical dialogue, we can recognize that each incremental improvement in our ability to address the range of potential demand comes with a sizable (actually exponential) increase in inventory requirements. This leads us to pursue a targeted service level at which we attempt to address a specific proportion of the full range of possible demand that balances the revenue goals of our organization with the cost of inventory.

The consequence of defining this service level expectation is that we must carry a certain amount of extra inventory, above the volume required to address our mean forecasted demand, to serve as a buffer against uncertainty. This safety stock, when added to the cycle stock required to meet mean periodic demand, gives us the ability to address most (though not all) fluctuations in actual demand while balancing our overall organizational goals.

The relationship between cycle stock and safety stock in addressing periodic demand

Calculating the Required Safety Stock Levels

In the classic Supply Chain literature, safety stock is calculated using one of two formulas that address uncertainty in demand and uncertainty in delivery. As our focus in this article is on demand uncertainty, we could eliminate the consideration of uncertain lead times, leaving us with a single, simplified safety stock formula to consider:

Safety Stock = Ζ * √PCT * σD

In a nutshell, this formula explains that safety stock is calculated as the average uncertainty in demand around the mean forecasted value (σD) multiplied by the square root of the duration of the (performance) cycle for which we are stocking (√PCT) multiplied by a value associated with the portion of the range of uncertainty we wish to address (Ζ). Each component of this formula deserves a little explanation to ensure it is fully understood.

In the previous section of this article, we explained that demand exists as a range of potential values around a mean value which is what our forecast generates. If we assume this range is evenly distributed around this mean, we can calculate an average of this range on either side of the mean value. This is known as a standard deviation. The value σD, also known as the standard deviation of demand, provides us with a measure of the range of values around the mean.

Because we have assumed this range is balanced around the mean, it turns out that we can derive the proportion of the values in this range that exist some number of standard deviations from that mean. If we use our service level expectation to represent the proportion of potential demand we wish to address, we can back into the number of standard deviations in demand that we need to consider as part of our planning for safety stock. The actual math behind the calculation of the required number of standard deviations (known as z-scores as represented in the formula as Ζ) required to capture a percentage of the range of values gets a little complex, but luckily z-score tables are widely published and online calculators are available. With that said, here are some z-score values that correspond to some commonly employed service level expectations:

Service Level Expectation Ζ (z-score)
80.00% 0.8416
85.00% 1.0364
90.00% 1.2816
95.00% 1.6449
97.00% 1.8808
98.00% 2.0537
99.00% 2.3263
99.90% 3.0902
99.99% 3.7190

Finally, we get to the term that addresses the duration of the cycle for which we are calculating safety stock (√PCT). Putting aside why it is we need the square root calculation, this is the simplest element of the formula to understand. The PCT value represents the duration of the cycle for which we are calculating our safety stock. The division by T is simply a reminder that we need to express this duration in the same units as those used to calculate our standard deviation value. For example, if we were planning safety stock for a 7-day cycle, we can take the square root of 7 for this term so long as we have calculated the standard deviation of demand leveraging daily demand values.

Demand Variance Is Hard to Estimate

On the surface, the calculation of safety stock analysis requirements is fairly straightforward. In Supply Chain Management classes, students are often provided historical values for demand from which they can calculate the standard deviation component of the formula. Given a service level expectation, they can then quickly derive a z-score and pull together the safety stock requirements to meet that target level. But these numbers are wrong, or at least they are wrong outside a critical assumption that is almost never valid.

The sticking point in any safety stock calculation is the standard deviation of demand. The standard formula depends on knowing the variation associated with demand in the future period for which we are planning. It is extremely rare that variation in a time series is stable. Instead, it often changes with trends and seasonal patterns in the data. Events and external regressors exert their own influences as well.

To overcome this problem, supply chain software packages often substitute measures of forecast error such as the root mean squared error (RMSE) or mean absolute error (MAE) for the standard deviation of demand, but these values represent different (though related concepts). This often leads to an underestimation of safety stock requirements as is illustrated in this chart within which a 92.7% service level is achieved despite the setting of a 95% expectation.

Required stocking only achieving a 92.7% service level when built using mean absolute error against a 95% service level goal

And as most forecasting models work to minimize error while calculating a forecast mean, the irony is that improvements in model performance often exacerbate the problem of underestimation. It’s very likely this is behind the growing recognition that although many retailers work toward published service level expectations, most of them fall short of these goals.

Where Do We Go from Here and How Does Databricks Help?

An important first step in addressing the problem is recognizing the shortcomings in our safety stock analysis calculations. Recognition alone is seldom satisfying.

A few researchers are working to define techniques that better estimate demand variance for the explicit purpose of improving safety stock estimation, but there isn’t consensus as to how this should be performed. And software to make these techniques easier to implement isn’t widely available.

For now, we would strongly encourage supply chain managers to carefully examine their historical service level performance to see whether stated targets are being met. This requires the careful combination of past forecasts as well as historical actuals. Because of the cost of preserving data in traditional database platforms, many organizations do not keep past forecasts or atomic-level source data, but the use of cloud-based storage with data stored in high-performance, compressed formats accessed through on-demand computational technology — provided through platforms such as Databricks — can make this cost effective and provide improved query performance for many organizations.

As automated or digitally-enabled fulfillment systems are deployed — required for many buy online pick up in-store (BOPIS) models — and begin generating real-time data on order fulfillment, companies will wish to use this data to detect out-of-stock issues that indicate the need to reassess service level expectations as well as in-store inventory management practices. Manufacturers that were limited to running these analyses on a daily routine may want to analyze and make adjustments per shift. Databricks’ streaming ingestion capabilities provide a solution, enabling companies to perform safety stock analysis with near real-time data.

Finally, consider exploring new methods of generating forecasts that provide better inputs into your inventory planning processes. The combination of using Facebook Prophet with parallelization and autoscaling platforms such as Databricks has allowed organizations to make timely, fine-grained forecasting a reality for many enterprises. Still other forecasting techniques, such as Generalized Autoregressive Conditional Heteroskedastic (GARCH) models, may allow you to examine shifts in demand variability that could prove very fruitful in designing a safety stock strategy.

The resolution of the safety stock challenge has significant potential benefits for organizations willing to undertake the journey, but as the path to the end state is not readily defined, flexibility is going to be the key to your success. We believe that Databricks is uniquely positioned to be the vehicle for this journey, and we look forward to working with our customers in navigating it together.

Databricks thanks Professor Sreekumar Bhaskaran at the Southern Methodist University Cox School of Business for his insights on this important topic.

--

Try Databricks for free. Get started today.

The post How a Fresh Approach to Safety Stock Analysis Can Optimize Inventory appeared first on Databricks.

Wednesday, 22 April 2020

Announcing Spark + AI Summit Hackathon for Social Good

Data Teams Unite!

We’re excited to announce our first-ever virtual and global hackathon, where you’ll form data teams to help tackle climate change, the COVID-19 pandemic or issues unique to your local community.

Data scientists, engineers and analysts are invited to collaborate and innovate for social good in the Spark + AI Summit Hackathon for Social Good

Your challenge

Apply your ideas and data skills to help address real-world problems.

To participate in the hackathon, follow these steps:

  1. Register a data team (up to four participants) on the Hackathon for Social Good website.
  2. Build an application or create a compelling notebook of your analysis that allows end users to better understand data related to these issues. Your application or notebook should use data analysis, data science or machine learning technologies featured in the Spark + AI Summit.
  3. Submit your project, along with a video screencast describing the potential social good impact.

Unite for a cause

By participating in the Hackathon for Social Good, your team’s good work will go toward a noble cause. In addition to helping us understand the data around these issues, the three winning teams will be invited to direct a donation to a charity of their choice, with a combined value of $35,000. Winning projects will also be announced in the Spark + AI Summit keynote on June 24 and recognized during special Summit events.

The grand-prize-winning team will award a charity with a $20,000 donation, receive free training and VIP passes to the June 22–26 Spark + AI Summit as well as complimentary passes to a future Spark + AI Summit.

Bring your best ideas to the biggest issues

When planning your hackathon project, we encourage you to focus on one of these three issues:

  1. Provide greater insights into the COVID-19 pandemic: Various COVID-19 data sets are now available on Databricks, Kaggle and GitHub. Use these sets — and other public sources — to surface insight into correlations, causes or potential solutions.
  2. Reduce the impact of climate change: Write an application or perform an analysis on the causes of or solutions to climate change.
  3. Drive social change in your community: What challenges do you see where you live and work? Check out a local city data set and inspire change close to home.

For details, including suggested data sets and complete submission and participation requirements, please visit the Hackathon for Social Good website.

Here’s what you need to know about timing:

Submissions:                April 22–June 12
Judging:                          June 15–19
Winners announced: June 24

If you have questions or comments, reach us at hackathon@databricks.com.

We can’t wait to see your projects.

START HACKING!

--

Try Databricks for free. Get started today.

The post Announcing Spark + AI Summit Hackathon for Social Good appeared first on Databricks.

Your Complete Guide to a Obtaining a 360 Customer View

If you’ve landed here, chances are you already know you want a 360-customer view for all the customer data in your organization.Cutting right to the chance, this blog post will help you:Identify Types of 360 Customer ViewsChallenges with Obtaining Consolidated Customer ViewsSolutions and Best Practices for Getting This ViewBenefits of 360 Customer ViewLet’s get started!Understanding 360 Customer View and The Types Of Information You Need  Simply put, a 360-customer view is a consolidated collection of your customer’s information gathered from multiple sources to give you an overall picture of your customer’s interactions with your organization at different phases of their journey.For example, signing up for an offer advertised on your social media platform followed later by a chat with your customer service department. All these interactions with different people and departments of your organization need to be captured to understand your customer’s journey which will later be instrumental in the predictive analysis (knowing what your customers may want in the future) and personalizing the customer experience.A 360-customer view provides several critical information that is essential when you want to create business growth, marketing or sales strategies. These are:Customer Submitted Data: While common attributes such as name, phone number, addresses are ...


Read More on Datafloq

Blockchain and Big Data: The Perfect Duo for Data Integrity

When it comes to blockchain, people perhaps could only relate it to cryptocurrencies. However, blockchain is not limited to cryptocurrencies but has rather developed enterprises that get a grip with other applications in the industry. Undoubtedly, blockchain and big data project to be one of the emerging technologies most companies are looking to adopt. Both of these technologies will transform how companies run their businesses. As a single entity both blockchain and big data might not be as useful as it depicts to be. But when combined: they could be powerful tools. Some may say, they’re perfect for each other. Here’s what Chris Neimeth, COO of NYC Data Science Academy has to say,“Big Data is an incredibly profitable business, with revenues expected to grow to $203 billion by 2020. The data within the blockchain is predicted to be worth trillions of dollars as it continues to make its way into banking, micropayments, remittances, and other financial services. In fact, the blockchain ledger could be worth up to 20% of the total big data market by 2030, producing up to $100 billion in annual revenue.”Blockchain and big dataAccording to Techjury, worldwide spending on blockchain projects to reach USD 11.7 billion by ...


Read More on Datafloq

Building a Modern Clinical Health Data Lake with Delta Lake

The healthcare industry is one of the biggest producers of data. In fact, the average healthcare organization is sitting on nearly 9 petabytes of medical data. The rise of electronic health records (EHR), digital medical imagery, and wearables are contributing to this data explosion. For example, an EHR system at a large provider can catalogue millions of medical tests, clinical interactions, and prescribed treatments. And the potential to learn from this population scale data is massive. By building analytic dashboards and machine learning models on top of these datasets, healthcare organizations can improve the patient experience and drive better health outcomes. Here are few real-world examples:

Real-world examples of healthcare organizations leveraging analytics and machine learning with their health data to improve the patient experience and drive improved outcomes.

Preventing Neonatal Sepsis


Learn more

Real-world examples of healthcare organizations leveraging analytics and machine learning with their health data to improve the patient experience and drive improved outcomes.

Early Detection of Chronic Disease


Learn more

Real-world examples of healthcare organizations leveraging analytics and machine learning with their health data to improve the patient experience and drive improved outcomes.

Tracking Disease Physiology Across Populations


Learn more

Real-world examples of healthcare organizations leveraging analytics and machine learning with their health data to improve the patient experience and drive improved outcomes.

Preventing Claims Fraud and Abuse


Learn more

Top 3 Big Data Challenges for Healthcare Organizations

Despite the opportunity to improve patient care with analytics and machine learning, healthcare organizations face the classical big data challenges:

  • Variety – The delivery of care produces a lot of multidimensional data from a variety of data sources. Healthcare teams need to run queries across patients, treatments, facilities and time windows to build a holistic view of the patient experience. This is compute intensive for legacy analytics platforms. On top of that, 80% of healthcare data is unstructured (e.g. clinical notes, medical imaging, genomics, etc). Unfortunately, traditional data warehouses, which serve as the analytics backbone for most healthcare organizations, don’t support unstructured data.
  • Volume – some organizations have started investing in health data lakes to bring their petabytes of structured and unstructured data together. Unfortunately, traditional query engines struggle with data volumes of this magnitude. A simple ad-hoc analysis can take hours or days. This is too long to wait when adjusting for patient needs in real-time.
  • Velocity – patients are always coming into the clinic or hospital. With a constant flow of data, EHR records may need to be updated to fix coding errors. It’s critical that a transactional model exists to allow for updates.

As if this wasn’t challenging enough, the data store must also support data scientists who need to run ad-hoc transformations, like creating a longitudinal view of a patient, or build predictive insights with machine learning techniques.

Fortunately, Delta Lake, an open-source storage layer that brings ACID transactions to big data workloads, along with Apache SparkTM can help solve these challenges by providing a transactional store that supports fast multidimensional queries on diverse data along with rich data science capabilities. With Delta Lake and Apache Spark, healthcare organizations can build a scalable clinical data lake for analytics and ML.

In this blog series, we’ll start by walking through a simple example showing how Delta Lake can be used for ad hoc analytics on health and clinical data. In future blogs, we will look at how Delta Lake and Spark can be coupled together to process streaming HL7/FHIR datasets. Finally, we will look at a number of data science use cases that can run on top of a health data lake built with Delta Lake.

Using Delta Lake to Build a Comorbidity Dashboard

To demonstrate how Delta Lake makes it easier to work with large clinical datasets, we will start off with a simple but powerful use case. We will build a dashboard that allows us to identify comorbid conditions (one or more diseases or conditions that occur along with another condition in the same person at the same time) across a population of patients. To do this, we will use a simulated EHR dataset, generated by the Synthea simulator, made available through Databricks Datasets (AWS | Azure). This dataset represents a cohort of approximately 11,000 patients from Massachusetts, and is stored in 12 CSV files. We will load the CSV files in, before masking protected health information (PHI) and joining the tables together to get the data representation we need for our downstream query. Once the data has been refined, we will use SparkR to build a dashboard that allows us to interactively explore and compute common health statistics on our dataset.

Example clinical health data lake architecture, demonstrating how Delta Lake can improve the exploration and analysis of large volumes of clinical data.

This use case is a very common starting point. In a clinical setting, we may look at comorbidities as a way to understand the risk of a patient’s disease increasing in severity. From a medical coding and financial perspective, looking at comorbid diseases may allow us to identify common medical coding issues that impact reimbursement. In pharmaceutical research, looking at comorbid diseases with shared genetic evidence may give us a deeper understanding of the function of a gene.

However, when we think about the underlying analytics architecture, we are also at a starting point. Instead of loading data in one large batch, we might seek to load streaming EHR data to allow for real-time analytics. Instead of using a dashboard that gives us simple insights, we may advance to machine learning use cases, such as training a machine learning model that uses data from recent patient encounters to predict the progression of a disease. This can be powerful in an ER setting where streaming data and ML can be used to predict the likelihood of a patient improving or declining in real-time.

In the rest of this blog, we will walk through the implementation of our dashboard. We will first start by using Apache Spark and Delta Lake to ETL our simulated EHR dataset. Once the data has been prepared for analysis, we will then create a notebook that identifies comorbid conditions in our dataset. By using built-in capabilities in Databricks (AWS | Azure), we can then directly transform the notebook into a dashboard.

ETLing Clinical Data into Delta Lake

To start off, we need to load our CSV data dump into a consistent representation that we can use for our analytical workloads. By using Delta Lake, we can accelerate a number of the downstream queries that we will run. Delta Lake supports Z-ordering, which allows us to efficiently query data across multiple dimensions. This is critical for working with EHR data, as we may want to slice and dice our data by patient, by date, by care facility, or by condition, amongst other things. Additionally, the managed Delta Lake offering in Databricks provides additional optimizations, which accelerate exploratory queries into our dataset. Delta Lake also future-proofs our work: while we aren’t currently working with streaming data, we may work with live streams from an EHR system in the future, and Delta Lake’s ACID semantics (AWS | Azure) make working with streams simple and reliable.

Our workflow follows a few steps that we show in the figure below. We will start by loading the raw/bronze data from our eight different CSV files, we will mask any PHI that is present in the tables, and we will write out a set of silver tables. We will then join our silver tables together to get an easier representation to work with for downstream queries.

Loading our raw CSV files into Delta Lake tables is a straightforward process. Apache Spark has native support for loading CSV, and we are able to load our files with a single line of code per file. While Spark does not have in-built support for masking PHI, we can use Spark’s rich support for user defined functions (UDFs, AWS | Azure) to define an arbitrary function that deterministically masks fields with PHI or PII. In our example notebook, we use a Python function to compute a SHA1 hash. Finally, saving the data into Delta Lake is a single line of code.

Loading raw CSV files into Delta Lake tables is a straightforward process.

Once data has been loaded into Delta, we can optimize the tables by running a simple SQL command. In our example comorbid condition prediction engine, we will want to rapidly query across both the patient ID and the condition they were evaluated for. By using Delta Lake’s Z-ordering command, we can optimize the table so it can be rapidly queried down either dimension. We have done this on one of our final gold tables, which has joined several of our silver tables together to achieve the data representation we will need for our dashboard.

Example use of Delta Lake’s Z-ordering command, to optimize the table so it can be rapidly queried down by either dimension.

Building a Comorbidity Dashboard

Now that we have prepared our dataset, we will build our dashboard allowing us to explore comorbid conditions, or more simply put, conditions that commonly co-occur in a single patient. Some of the time, these can be precursors/risk factors, for example, high blood pressure is a well known risk factor for stroke and other cardiovascular diseases. By discovering and monitoring comorbid conditions, and other health statistics, we can improve care by identifying risks and advising patients on preventative steps they can take. Ultimately, identifying comorbidities is a counting exercise! We need to identify the distinct set of patients who had both condition A and condition B, which means it can be done all fully in SQL using Spark SQL. In our dashboard, we will follow a simple three step process:

  1. First, we create a data frame that has conditions, ranked by the number of patients they occurred in. This allows the user to visualize the relative frequency of the most common conditions in their dataset.

    Example use of Spark SQL to create a data frame that has health conditions, ranked by the number of patients they occurred in. This allows the user to visualize the relative frequency of the most common conditions in their dataset.

  2. We then give the user widgets (AWS | Azure) to specify two conditions they are interested in comparing. By using Spark SQL, we identify the full set of patients that the condition occurred in.

    Example use of Spark SQL to create widgets that specify two health conditions for comparison.

  3. Since we do this in Spark SQL using SparkR, we can easily collect the count of patients at the end and use a χ2 test to compute significance. We print whether or not the association between the two conditions is statistically significant.

While a data scientist who is rapidly iterating to understand what trends lie in their dataset may be happy working in a notebook, we will encounter a number of users (clinicians, public health officials and researchers, operations analysts, billing analysts) who are less interested in seeing the code underlying the analysis. By using the built-in dashboarding function, we can hide the code and focus on the visualizations we’ve generated. Since we added widgets into our notebook, our users can still provide input to the notebook and change which diseases to compare.

Example use of SparkR in SQL Spark to collect a count of patients and apply a χ2 test to compute significance.

Get Started Building Your Clinical Data Lake

In this blog, we laid down the fundamentals for building a scalable health data lake with Delta Lake and a simple comorbidity dashboard. To learn more about using Delta Lake to store and process health and clinical datasets:

--

Try Databricks for free. Get started today.

The post Building a Modern Clinical Health Data Lake with Delta Lake appeared first on Databricks.

How Can Small Businesses Utilize The Power Of AI

Artificial intelligence is revolutionizing industries, departments, and the way we live our daily lives. Innovative technologies are emerging and changing the way we interact with our environment while small businesses are certainly not left behind. They have the opportunity to seize the very best of AI and improve their performance exponentially. In fact, studies have shown that almost 30% of small and medium-sized businesses believe that AI will have the greatest impact on their companies. Here we will take a closer look at exactly how.

1. Automating reporting processes

To be able to effectively make better business decisions, small businesses need a clear overview of the data they monitor and report on. Traditionally, spreadsheets were enough but as more information is gathered each day, the need for professional tools that can process enormous volumes of information, automatically report on data and business processes – is becoming a standard. Tracking KPIs and sending reports on a daily, weekly, or monthly level enable small businesses to focus on the data itself, without performing tedious tasks of building spreadsheets and trying to manually process different business touchpoints.

2. Relieving the customer service department

Many small businesses deal with customers but the quality and speed of answering questions ...


Read More on Datafloq

Tuesday, 21 April 2020

The Best Practices for Instilling the Enterprise Data Management for Your Business

Enterprise Data Management or EDM is the capability of an organization to govern, integrate, secure, and distribute data from multiple data sources. EDM also comprises the ability to transfer data among subsidiaries, partners, applications, and processes in a perfect and secure way. Successful EDM can only be a result of crystal clear understanding of data and implementation of an intelligent EDM strategy.Components of Enterprise Data Management· Data Governance – Data governance encompasses the guidelines, policy enforcement, and processes used to safeguard the quality, integrity, and security of data in an organization.· Data Integration – Data integration refers to collecting and consolidating a business’s diverse data into one, accessible place. Propagation, virtualization, consolidation, and federation are the different types of data integration.· Data Security – Another vital component, data security typically refers to the methods and approaches implemented to ensure that the data is protected whether in transit or stored.· Master Data Management – Master Data Management refers to tools or applications used as part of an enterprise data ...


Read More on Datafloq

Pen-Tests Improve Enterprise Security

Although organizations strengthen their security ties, yet attackers work harder. However, a pen testing company suggests that most of the organizations are making progress in securing their systems against cyber attacks. They gather aggregated data from penetration tests and red team engagements to highlight the vulnerabilities that companies rectify to secure their networks, systems, and applications. With the help of their external and internal assessments, pen testing companies highlight that although organizational networks continue to depict multiple weaknesses, thus, attackers may have a hard time identifying and exploiting them from outside the network. Security organizations face big huge challenges that result in complex security hygiene, patch management, password quality and lack of visibility. Thus, the attackers are forced to change their tactics and employ malware-free, approaches to conceal malicious activities. It is due to the exploitation of the environment, now organizations need tools and technology to observe normal system functions to determine if they are being used maliciously. The pen-tests engagements are performed to see all the similarities it could find across enterprise networks. Data from hundreds of pen-tests showed that accounts with weak and easily identifiable passwords are one of the biggest problems for most of the organizations. ...


Read More on Datafloq

Indian engineer develops rapidly producible Covid-19 ventilators

Citadel Research and Solutions has developed cost-effective ventilators that were tested successfully by healthcare professionals on the patients.

Monday, 20 April 2020

The Impact of Blockchain Technology on Various Industries Expected in This Decade

When we talk about blockchain technology, the laymen audience tends to automatically associate it with tokens or cryptocurrency. The reality is that blockchain has countless possible uses that have direct and concrete applications in a large number of industries, both in the public and private spheres. To cite just a few examples, it can be implemented in sectors such as financial, insurance, digital advertising, supply chain, health, education, Internet of things, etc. It also has the ability to be used to generate positive social impact, as in the production of food or medicine.Mediledger, for example, is a proposal based on blockchain technology with the intention of generating end-to-end traceability in the drug production process. On the other hand, Grassroots is a project that seeks to make beef meat traceable. Also, many companies in different countries are exploring solutions, such as Hyperledger, to allow consumers to analyse the production cycle of a food.Another specific case is from the donation industry. The traceability that blockchain technology can provide makes the process more transparent to all parties involved. This is being pioneered by the likes of Givetrack, Humaniq, and Circles of Angels. It can also be applied as a solution to the unbanked ...


Read More on Datafloq

Can coronavirus crisis change Indian science for good?

A frenetic behind-the-scenes race is underway to find and quickly bring to market technological solutions to the Covid-19 crisis

Sunday, 19 April 2020

How Big Data Can Help In A Pandemic

The past few years have witnessed the rise of big data, which has upended established ways of doing business while promising to usher in a new era of digitally-defined analytics. While relatively few people were thinking about the role of big data when it comes to fighting health crises just a few years ago, the continued spread of COVID-19 has led many to ask how big data initiatives can help in a pandemic. It’s becoming increasingly clear that our ability to generate, record, and analyze data is an essential part of our pandemic response strategy.From Israel to China, nation-states are embracing big data surveillance like never before to resist the COVID-19 pandemic. Contact tracing is importantWhen it comes to responding to a pandemic like COVID-19, contact tracing is of immense importance. Contact tracing involves tracking down the individuals that you came into contact with after being exposed to a contagion like COVID-19; if we’re incapable of determining who the infected were in contact with once they became contagious, we’re incapable of preventing the spread of the infection. While contact tracing has been around for many years, it’s becoming better than ever before with a little help from big data.Smartphone apps ...


Read More on Datafloq

Saturday, 18 April 2020

Blockchain Is a Key Technology for the Development of Internet of Things (IoT) Solutions

Blockchain has gained enormous attention since it was first launched as a form of distributed ledger technology back in 2011 - and not only for its cryptocurrency potential. From revolutionising the music, real estate, logistics, recruitment and healthcare industries, to upending the way we receive funds and pay bills, blockchain has been hailed as transformational on many fronts. But it's the convergence of blockchain and the Internet of Things (IoT) that is on the agenda for a great number of companies, as it slowly dawns on us that blockchain could provide the solution to perhaps the greatest challenge of IoT: third generation security.For those of us who aren’t as technologically-savvy as they’d like to be, the IoT describes the system of interrelated computing devices, mechanical and digital machines, or other objects that are able to transfer data over a network, with no human-to-human or human-to-computer interaction required. The IoT describes the interconnection of computing devices embedded in everyday objects, like connected security systems, cars, electronic appliances, alarm clocks, speaker systems, baby monitors and so forth. Thanks to the development of incredibly cheap computer chips and the pervasiveness of wireless networks, there are now billions of physical devices and objects ...


Read More on Datafloq

Hackers are a busy lot in these lockdown days

With many working from home, safety processes may not be as tight as those in offices, say experts

Friday, 17 April 2020

Mass Hysteria Surrounds COVID-19 Tracking Apps, and We Are Sleepwalking to State Surveillance



The world is in crisis, and whenever a crisis occurs, drastic measures could be taken by those involved. In this particular case, the Coronavirus has the entire world in its grip and governments around the world are struggling to contain the health and economic effects of the virus.

The lockdown in place in most countries seems to be effective in limiting the spread of the virus. The same applies to social distancing measures now in place all around the world. The good thing about these measures is that they will disappear once the virus is gone, though it might take some time. Eventually, the lockdown will be lifted and, although we will experience the ‘1,5-meter economy’ for some time, that will be lifted as well.

However, in many countries around the world, new, unproven, measures are being considered or are already in place. I am talking about COVID-19 tracking apps that are being developed all around the world. The list of countries is growing rapidly:


The Netherlands: a public call for help resulted in hundreds of companies offering their support in developing a Corona-tracking app. It is the objective of the government to get an app up and running within weeks;
France and Germany: ...


Read More on Datafloq

Crypto and Real Estate Soulmates, Made for Each Other


Regardless of whether you’re selling, buying, or tokenizing real estate, it is undeniable that the real estate/property market and blockchain are intricately interconnected. As purchasing real estate using bitcoin has become an option, the broader potential of bitcoin to revolutionize the industry is becoming apparent. The confluence of crypto and the real estate market is no longer fanciful speculation, but a burgeoning sector that has already started to reap dividends. 

Real estate evolved: buying to tokenizing

Not many assets are more illiquid than real estate. The interbreeding between real estate and blockchain can be observed as far back as 2013 when Ragnar Lifthrasir initiated the International Blockchain Real Estate Association (IBREA). IBREA, as a fundamentally academic resource, did not garner much attention in its own right. However, it demonstrated that cross-sector interest was present and brewing. This mutual curiosity and interest took some time to evolve into the swapping of coins and keys – when this finally happened, Bitpay became one of the market pioneers.

During January of 2017, Bitpay revealed that in one of the real estate purchases in which the organization had been involved, the seller raked in an extra 1.3 million USD. The property was valued at approximately 4 million ...


Read More on Datafloq

UPS Committed to Electric Vehicles on an AI Platform

By AI Trends Staff

UPS, the logistics and delivery company, is spearheading a project in England to explore how AI systems can optimize the charging of electric fleet vehicles, and help integrate onsite renewable energy resources at vehicle depots.

The EV Fleet-Centered Local Energy Systems (EFLES) project is scheduled to start in May at the UPS depot in the Camden borough of London, according to a recent account in electrive. The UK Power Networks Services will provide oversight, while the smart battery and EV-charging software provider Moixa will contribute its GridShare smart AI platform to manage solar, storage and charging assets.

“We have the global expertise, smart-charging infrastructure and resources to host this first-of-a-kind test bed at our Camden facility,” stated UPS sustainable development coordinator Claire Thompson-Sage. “This project will build on our EV infrastructure technology to help develop a holistic local energy system.”

Claire Thompson-Sage, UPS Sustainable Development Coordinator

The GridShare software helps track hundreds of data sources for energy prices, power demand, weather conditions and more, to help determine which charging times are less expensive and which mix of renewable energy makes the most sense at any given point in time.

The ERLES project is the next stage in the UPS partnership with Arrival, the UK-based “generation 2” electric vehicle manufacturer, which developed its newest vehicle with UPS.

UPS recently placed an order for 10,000 electric vehicles from Arrival, to be delivered from 2020 to 2024, according to a recent press release issued by Arrival.

UPS co-developed the Generation 2 vehicles with Arrival, which employed a new method of assembly using low capital, low-footprint micro-factories located to serve local communities and be profitable making thousands of units. The UPS partnership with Arrival was first announced in 2016.

“UPS has been a strong strategic partner of Arrival, providing valuable insight to how electric delivery vans are used on the road and how they can be optimized for drivers, stated Denis Sverdlov, founder and CEO of Arrival. “Together our teams have been creating bespoke [custom] electric vehicles, based on our flexible skateboard platforms, that meet the end-to-end needs of UPS from driving, loading/unloading, depot and back office operations.”

Denis Sverdlov, founder and CEO of Arrival

Carlton Rose, President of UPS Global Fleet Maintenance & Engineering, stated, “Our investment and partnership with Arrival is directly aligned with UPS’s transformation strategy, led by the deployment of cutting-edge technologies.These vehicles will be among the world’s most advanced package delivery vehicles, redefining industry standards for electric, connected and intelligent vehicle solutions.”

UPS Has Had Long Commitment to Electric Vehicles

UPS had 1,000 electric vehicles in its fleet of 112,000 vehicles two years ago. The cost of the vehicle new was found to be no more than regular diesel vehicles, because the cost of electric batteries plummeted 80 percent in six years, according to a UPS press release from April 2018.The electric transporters are expected to create additional value of UPS in operational savings and routing efficiency.

In the U.S., UPS has been working with Workhorse to develop an electric transport vehicle. The target at the outset was a range of 100 miles, with similar procurement costs as an internal combustion motor. The founder and CEO of Workhorse, Steve Burns, late last year bought the Lordstown, Ohio electric vehicle plant from General Motors, through a company he set up to execute the transaction, Lordstown Motors. He has said he wants to build electric pickup trucks for “business and government customers” and has decided the name of the first model will be: Endurance.

Financially, Workhorse has faced some challenges, losing $38 million in 2019 and having little sales in late 2019, according to a recent account in The Verge. Workhorse will own 10 percent of Lordstown Motors, and license to it the intellectual property related to the planned W-15 electric pickup truck. Burns will transfer 6,000 pre-orders for the truck to Lordstown. He is searching for financing, saying he needs $300 million to start product in a year. He plans to run a union shop and produce 500,000 vehicles per year, double the number of Cruze sedans GM made at the plant.

In the case of these new trucks, UPS worked closely with a supplier, Workhorse, to redesign the trucks “from the ground up,” stated Scott Phillippi, UPS’s senior director of maintenance and engineering. Phillippi expects the new design will reduce the truck’s weight by some 1,000 pounds, compared with a diesel or gas-powered vehicle. That plus better batteries will give the truck an electric range of around 100 miles, enough for most routes in and around cities.

Read the source articles and releases at electrive, a UPS press release on the 10,000 vehicle order from Arrival,  and in The Verge., and a UPS press release from April 2018.

Jaywalking and AI Autonomous Cars

By Lance Eliot, the AI Trends Insider

Being from California, I remember one of the first times that I visited New York City (NYC) and made the mistake of renting a car to get around the famous metropolis. I had figured that driving a car around the avenues and streets would give me a good sense of how the city that never sleeps was laid out and where the most notable restaurants, bars, and shops could be found.

Turns out that I mainly discovered how much New Yorkers seemed to delight in jaywalking.

It was as though there weren’t any rules against jaywalking.

Want to cut across the street and get over to that popular hangout, no need to walk down to a crosswalk, instead just make your way by walking into traffic. In most cases, the jaywalker didn’t even run. One might almost think that you would dart rather than meander, but these fearless jaywalkers tended to take their time.

I also found out about the techniques involved in making a devoted stare or gaze that appeared to be a local custom.

In some cities, the jaywalker purposely does not make eye contact with the car drivers, seemingly acting as though the car drivers don’t exist.

Or, maybe by making eye contact it would become a duel to see who looked away first, and the loser perhaps has to back-down from the standoff.

In any case, my experience was that the jaywalkers in NYC loved to give the car drivers a straight eye.

This might be the same kind of thing you’d do when you encounter a wild animal in the woods. Given them a strong stare might say that you are mighty and the animal should not try to take you on. Some of the car drivers that were locals or that were used to the local customs would often give a stern stare back. On a few occasions, it would get really testy and the jaywalker would wave an arm and act as though they might try to slay the dragon of a car coming down the street.

I admit that after I turned in the rental car and became more of a traditional pedestrian on my visits to NYC, I adopted the jaywalking habit.

This was especially so because during one of my initial forays as a pedestrian there, I was walking with a colleague that was a native New Yorker, and when I attempted to walk down to a crosswalk, rather than taking the shortcut of jaywalking, he almost came out of his skin at my legal abiding approach.

Are you nuts, he asked or demanded incredulously?

Walk half a block down, cross the street at a light, and walk a half block back up, just to get to something that you could make a beeline to?

I regret that perhaps it gave another black eye in his NYC mindset of my being from the West Coast.

He even justified the jaywalking in a manner that perhaps most would not.

He insisted that it was actually more dangerous to cross at a marked crosswalk, at least in NYC, than it was to jaywalk. I doubt that he had any actual statistics to back the claim, but it certainly sounded convincing. He had me watch the cars turning at a busy corner and pointed out that I would seemingly be more likely to get run over there. With a jaywalking maneuver, he emphasized that I could pick my own choosing of when and where to cross, presumably therefore somehow being a much safer adventure than depending upon an actual marked crosswalk.

Where I grew up in California, jaywalking was generally frowned upon and only undertaken as some kind of last resort.

If you had a broken leg and could not walk all the way to a corner, okay, maybe you could do a jaywalk, but only if the street was absolutely clear of traffic. No “frogger” kind of playing in my neighborhood.  I remember my parents even hinting that the cops would likely be driving down the street just as I might try to jaywalk. This made me envision a life of sitting in prison due to having gotten caught red-handed doing a jaywalk. I wondered whether I would do hard time and also if I might ever be able to make parole due to the seriousness of my transgression against society’s rules.

One time, a relative from New York came out to visit and noticed that some of the streets in my neighborhood had a posted sign that indicated jaywalking was prohibited.

First, he laughed at the sign and declared it to be a total waste of taxpayer money.

Second, he interpreted the sign to imply that wherever there wasn’t a similar sign, it meant that you could legally jaywalk, and do so as much as your heart might desire. I tried to explain that jaywalking was generally outlawed locally and the purpose for the signs was to highlight the law, particularly in places where it was known that people tended to jaywalk, even though they weren’t supposed to do so, and serve as a reminder of the law.

Styles of Jaywalking

Since I had not seen much jaywalking growing up, it was fascinating to watch it occur while I had various stays in NYC.

I noticed for example that the time of day seemed to make a difference in terms of the volume and nature of the jaywalking.

Mornings, when pedestrians were trying to get to work, often stoked a lot of jaywalking, perhaps to try and get to work promptly and minimize the time required to get to the office.

There was also the amount of traffic that played a role in the jaywalking. If the traffic on a given street was completely backed-up and stuck, jaywalkers would in droves weave in and around the cars, doing so without a care in the world since they perceived that the wild animals (the cars and car drivers) were jammed in place and couldn’t do much to run them over. As soon as a green light allowed the traffic to flow, the jaywalkers became more cautious and realized it was now “game on” in terms of trying to time when to best engage in jaywalking.

If a street had intermittent traffic, and if the cars that used the street considered it to be a kind of race track to quickly make some progress through the slew of blocks of NYC, the jaywalker had to be much more nimble and aware. Will that car that just turned onto the street be burning rubber and get to where you are going to jaywalk, reaching an intersecting point in the middle of the street just as you are halfway done with your jaywalk maneuver? These crazed drivers made it appear that they were not going to stop for anything and nor anyone. I don’t care if you had your pet elephant on a leash and were jaywalking with it, these mean looking and solemn minded drivers were willing to smash their car into whatever might be in the roadway. The road was there’s and no one dare suggest otherwise.

The weather also played a part in the jaywalking ritual.

Rainy days meant that the jaywalkers had an even greater incentive to jaywalk. Why waste time and get wet in the rain, when you can scoot across a street and do so quickly enough that perhaps rain drops themselves won’t touch you. The problem with the feverish effort to jaywalk in rain was the car drivers were likely to also be more crazed than usual. I suppose this was because the rain tended to hamper traffic and therefore the way to make-up for it was to speed and be a bit more careless of your driving. I realize you might assume it should be the opposite, namely you would slow down in the rain and be more careful, but that’s not often the choice that drivers seem to make (this almost seems like a universal constant!).

At times, I pondered the nuances of Mutually Assured Destruction (MAD), which you might remember was popularized during the Cold War era. When the two hunkering superpowers of the United States and the Soviet Union had their nuclear arms race, it was postulated that if either one attacked, the other would surely attack, and in the end they would both obliterate each other.

This came to mind as I watched some of the jaywalking duels in NYC.

An energetic jaywalker would enter into the street.

A zealous driver would gun their engine and seem to aim for the jaywalker.

Which would win the race?

I’m sure you are pleading that obviously the car will win, since a mere human is not going to have super powers to stop the car in its tracks. In that sense, certainly the car can always prevail by running over the human. You might think that’s what would happen. Instead, it was interesting that even the nuttiest of drivers seemed to realize that running over a jaywalker was not an advisable thing to do.

Presumably, the car driver might be thinking that they could possibly get prosecuted for running over a jaywalker. Or, maybe they were worried it would dent their beloved car. Or, they might be concerned that their insurance rates would get jacked sky-high. There’s also the possibility that the driver might not want to maim a fellow human being. Well, being realistic, I’m putting that on the bottom of the list of reasons why the drivers did not summarily run over the jaywalkers.

So, it was often a Mutually Assured Destruction kind of battle.

The jaywalker figured that the driver figured that hitting the jaywalker would not be a good thing to do. The driver figured that the jaywalker figured that getting hit by a car was not a good thing to have happen. Either way, if a physical connection was going to be made, it was a lousy outcome for both parties.

Some of the jaywalkers acted as though they had a special invisible shield that would protect them. They would walk across the street whenever they darned wished to do so. They seemed to believe that the drivers would ultimately acquiesce and not want to run over a jaywalker. Admittedly, this did seem to work a lot of the time.

When I mentioned to one of my NYC colleagues that the Mutually Assured Destruction won’t serve as a deterrent unless both parties are cognizant and aware of what is taking place, he shrugged it off. I was trying to explain that if say the driver is not paying attention to the road, and not especially cognizant of the presence of the jaywalker, the driver could ram into the jaywalker out of “ignorance” and the jaywalker’s expectation of being protected by MAD went out the window. The MAD approach only worked if the driver was truly paying attention to the road.

Driver Attention And Jaywalkers

In my estimation, a sizable chunk of drivers were not attuned to the presence of the jaywalkers.

This made sense since the drivers were having to contend with bigger game, such as large trucks trying to make deliveries and rapidly exiting and entering unexpectedly into the street and avenues. There were other crazed car drivers jockeying for position. There were often obstacles on the roadway such as a pallet of liquor bottles being delivered to a liquor store.

If you were a driver in that environment, which is a more suitable aspect to pay attention to?

The trucks and other cars are likely more harmful to you and your car. The solid obstacles on pallets could do some real damage to your car if you hit them.

A jaywalker?

Not the highest priority.

Furthermore, many of the drivers seemed to consider that a jaywalker did jaywalking at their own risk. In essence, the car driver did not have to pay attention to the jaywalkers because the jaywalkers were “required” to always make sure to avoid getting hit by a car. It was as though a flock of birds were flying around the cars. A driver shouldn’t have to watch out for the birds. The birds should be astute enough to not flap into a car. The jaywalkers were assumed to be hopefully as astute as a dumb bird.

Another factor involved sizing up the jaywalker.

How was the jaywalker dressed and what kind of look did they have?

If a driver saw a jaywalker that seemed like a seasoned New Yorker, it suggested that the jaywalker could take care of themselves and no further driver attention was needed. If the jaywalker looked like a wide-eyed tourist, well, this might present a problem because the “amateur” jaywalker might foul things up. The “professional” jaywalkers knew how to assiduously cross a street. Those out-of-town jaywalkers were bound to mess-up the delicate dance of true jaywalkers and NYC drivers.

I’m sure that when I drove my rental car, the jaywalkers could sense my out-of-town smell.

Fresh meat, easy pickings.

I was the type that they could jaywalk to their hearts content on. Indeed, when I saw a jaywalker, I tended to give them a wide berth. The seasoned NYC drivers in contrast would always relish getting within inches of the jaywalker, as though it was a sweet kiss of “you just made it” and the jaywalker should thank their lucky stars for surviving the jaywalking act (and bend down in reverence to the car driver).

When it got somewhat late at night, I observed that there would be a segment of jaywalkers that were a bit intoxicated, having visited their preferred pub for some after-work libation. This seemed to dampen their wits as jaywalkers. I’m betting that they would contest this claim and say that they were still on their toes. In any case, there were definitely more close-calls on the jaywalker versus car aspects. This might also be further fueled by the likelihood of having drivers that were now also somewhat drunk. A potent combination, having both slightly drunk jaywalkers and slightly drunk drivers.

Herd Mentality Of Jaywalking

In most cases, I observed individuals acting as jaywalkers.

This though was not always the case and there were frequently situations of multiple jaywalkers proceeding all at once. There was at times a herd mentality. If one of the jaywalkers went for it, the others were sure to follow. Now, this actually often made sense, since the first one likely found an opening to jaywalk and the others also perceived the same opening.

There were times though that the first jaywalker got the herd underway, not necessarily overtly, more so subliminally in that the other jaywalkers saw the first one make a move and opted to proceed too, but it turned out that the first jaywalker didn’t gauge things well. The first jaywalker might have gotten somewhat stranded in the street, not able to fully make it across the street just yet. Meanwhile, the herd that followed was also now stranded. You could see the look on their faces that they had assumed they could make it fully across the street and were befuddled and irked that the move had not been timed well.

There were some “leaders” of the pack that weren’t thinking at all about the rest of the herd. Therefore, they were not trying to find a big enough opening to get a dozen people across the street all at once. They were focusing on just themselves. In that case, sometimes the first mover made it across, but the others did not, and they had somehow assumed that if the first mover could do it, the rest of them could.

I suppose you could see this as a series of locking mechanisms that just happen to line-up precisely in a moment of time. The first mover has “calculated” that they can thin their way through the sporadic cars and make it across. It is though just a moment in time action. A split second later and the opportunity has vanished. Likewise, the alignment of the cars is not just a moment in time, but also a moment in space, as it were. The first mover at their position say halfway of the block, would have a different timing and clearance of an opening, versus if you were at a quarter way of the block.

The ones that got my heart pumping were situations in which two people were holding hands and opted to rush across the street together. As you can imagine, trying to get two people across on a precisely timed jaywalk is a lot more complex. If one of the two falters it can defeat the open window and you now have to figure out what to do. It was surprising at times to see how much connectedness was retained by the two.

In other words, two people are holding hands. This is obviously just a temporary connection in that their two hands are not glued to each other. They can separate their hands whenever they wish. And yet, in some cases, the pair would try to remain entangled, in spite of the danger of doing so. Just a mere dropping of their hands would allow them both to become free agents and more nimbly finish the jaywalk.

This though seemed to be the further most thought in some of their minds. It was as though the separating of their hands meant more to them than the chances of getting hit by a car. Was it true love that kept them together in that life risk move? Was it concern that the other one might feel abandoned and it could forever undermine their relationship? Maybe it was out of deep caring and the belief that by sticking together they could survive anything, including a crazed driver barreling down the street directly at them.

There’s another kind of coupling sometimes that occurs, involving a jaywalker that is jaywalking with their dog. The jaywalking human might be hand carrying the dog, having lifted the dog up and embracing the animal like you would carry a football. This makes sense in that having the dog walk on a leash is going to be much more uncontrollable as you make your way across the street. For those that don’t try to carry their dog, perhaps due to the weight and size of the dog, the leash approach can be quite dicey.

I remember seeing a man walking his dog that had a leash several feet long and as the man attempted to jaywalk, the dog tried to go in a different direction. This meant that the jaywalker was now several feet wide, if you consider the distance from him to his dog, making it much harder to nimbly get across the street. He pulled strenuously on the leash, nearly dragging the dog, as he desperately tried to bridge the chasm from one side of the street to the other.

In this case, he was somewhat strongly coupled because letting go of the leash would have produced perhaps even worse results. The dog might have scampered directly into traffic that otherwise the jaywalking human might have aided the dog in avoiding. All in all, I would say that any animal lovers would look upon these jaywalkers with some disdain, as it is one thing to put your own life into jeopardy and quite another to subject an innocent dog to the same kind of risk.

This reminds me too of a common refrain that my New York colleagues would use on me.

They would say that any jaywalker is making their own decisions and if they get hit, well, that’s their own doing. Why should the government tell them what they can and cannot do. It’s up to the individual to choose to jaywalk or not, and it is on the head of that jaywalker as to whether they risk their life and limb or not.

I don’t buy into this claim per se.

It seems to leave the car drivers out of the equation. If a car driver hits a jaywalker, it’s going to be a great deal of difficulty for the car driver, though yes I agree it is unlikely the car driver will be killed, but they could get injured. Furthermore, suppose the car driver is so anxious to avoid hitting a wayward jaywalker that the driver rams into another car? Now, you’ve got other people also enmeshed into the jaywalking effort.

There is also a chance that while the car driver is trying to avoid a jaywalker, the driver swerves and maybe hits other jaywalkers (I realize the view would be that’s on them, if you take the individualist free agent perspective), or might come onto the curb and hit pedestrians (one would argue those pedestrians were innocents).

Generally, a jaywalker can start a cascading series of events that ultimately lead to others getting injured or killed.

I therefore tend to reject the idea that a jaywalker is performing a “victimless” act, assuming that you don’t count the jaywalker as a victim, and contend that the jaywalker is potentially going to involve one or more car drivers, perhaps one or more passengers in those involved cars, maybe other jaywalkers, and potentially innocent pedestrians that were mindfully using the sidewalk.

Here’s another angle for you.

What about the children?

Children Learning About Jaywalking

I’ve seen jaywalkers holding the hand of a child or a group of children and trying to make a jaywalking attempt with them. Similar to my earlier point about coupling between two adults, in theory the coupling is loose since the hands can be disengaged readily. In the case of children, the danger obviously is that if the adult jaywalker does let go of the hands of the children involved, the children might not know what to do and get themselves into worse hot water.

I realize some would argue that of course the adult needs to hold the hands of the children and would accuse me of somehow suggesting that children should roam freely as jaywalkers. Let’s be serious, I’m not implying that children should be unescorted by an adult when jaywalking. The thing is that children should not be jaywalking at all.

I’ll probably get emails from some readers that will say that their children have “no choice” but to jaywalk and so which will it be, the children do so on their own or with an adult? I guess if there is really no other viable way to get to someplace other than jaywalking, yes, an adult jaywalker participant is the way to go. Is it really the case that there is no other viable way to get to the location other than jaywalking?

There are some that have said to me that it would require walking several added blocks and take another 15 minutes to get to the desired location, such as a school. Well, one has to then consider the ROI (Return on Investment) of walking those extra blocks and using those added 15 minutes, doing so presumably in a safer manner, versus the risks associated with doing the beeline jaywalking. Is there an appropriate risk/reward that says the added risk to the child makes the jaywalking act worthwhile?

One other qualm about involving children into jaywalking is the aspect that they essentially then come to believe that jaywalking is acceptable.

If they do jaywalking with an adult, it is a slippery slope that can readily assume they can do jaywalking on their own. Indeed, some children will happily go jaywalking to showcase to their parent that they are now sentient and their own agent and no longer need to have an adult aid them in the jaywalking. It is a kind of rites of passage.

The counter argument from some adults is that if they don’t show the child how to “properly” jaywalk, the odds are that the child is going to do jaywalking anyway at some point, and without having done so with a “responsible” adult, the child is going to be more prone to getting hurt when trying to go jaywalking based on no prior instruction. Some would say that having a head-in-the-sand viewpoint of being a parent that pretends jaywalking will never happen, will merely make the child more vulnerable than if you instead do a parent-child jaywalking effort with the child.

There are some aspects of the counter-argument that I do tend to side with. In the case of my own children, I did practice going jaywalking with them, which I did to point out how to do so and what to watch out for. This was done though on a selective basis and only as a means to aid them in being prepared in case jaywalking was needed at some point. I tried to make clear cut that jaywalking was considered inappropriate and that the instruction was not meant to open the sport of jaywalking to them.

This also gets to the core of an aspect about children and child rearing. For those of you with children, you’ve likely been torn about whether to show or explain something to the child, for which you wonder whether you are introducing them to something that will spur them to do the thing you are trying to showcase should not be done. The classic is “don’t put your hand in a stove top burner,” which could backfire in that the child might not have thought to do so, and now they are curious to try it, because you made such a big deal about it.

In any case, another facet of jaywalking is the possibility of adult jaywalkers, child jaywalkers, and combinations of both adult and child aged jaywalkers.

There’s the special twist of the jaywalker that drops something while in the act of jaywalking.

The Dropped Item As Jaywalking Factor

I saw a jaywalker that was carrying his coat as he darted across the street. The street was slightly wet from leftover rain. The person slipped while running across the street. As he regained his balance, he dropped his coat. At this point, his presumed prior calculated time to get across the street had been used up. A car was fast approaching. Should he pick-up his coat, which would take a precious second or two, and tempt fate with the ongoing car, or should he abandon the coat and safely get to the sidewalk.

Which is better, a coat that perhaps gets trampled by a moving car, and for which you can go out the street once the car has passed, readily pick-up the coat, and maybe get it dry-cleaned to fix it up, or do you bend over while in the middle of the street and watch the ongoing car like it is a bull charging at you in Pamplona?

The answer seemed to be that nearly every time this kind of dropping action happened, the person opted to try to pick-up the dropped item.

Was this out of a sense of personal affiliation with the dropped item?

Maybe the coat had been in the family for many generations and was revered heirloom. Or due to the value? Perhaps it was an expensive coat from a top-end retailer. Or, could it be that the jaywalker was worried that the car driver might swerve to avoid the dropped item and therefore get into a wreck by having left it in the street? I doubt this is the first thought that goes through the mind of the jaywalker that dropped an item.

A car driver that witnesses a jaywalker dropping an item will need to figure out whether the jaywalker is going to try to stay in the street to retrieve it, or leave it there for later retrieval, or perhaps do some other kind of action now that the dropped item is there. The driver also needs to anticipate that maybe some other potential jaywalker might enter into the street to rescue the item for the first jaywalker that dropped the item. Whatever item has been dropped, the driver also needs to decide whether to try to brake before hitting it, if there is a chance of hitting it, or maybe try to straddle the item, or take some other kind of evasive driving action.

I’ve been so far primarily describing the jaywalkers and you’ll notice that I’ve now started to shift focus toward the car drivers and the act of jaywalking pedestrians.

In terms of the drivers, there are drivers that know the jaywalking game and play it to the finest detail. There are the drivers that are driving while distracted and so inadvertently can be more menacing for a jaywalker. There are the drivers that are drunk or otherwise somewhat incapacitated and therefore are impaired while playing the jaywalking game.

There are also the vendetta drivers.

Vendetta Drivers On The Roads

I cannot say for sure that vendetta drivers really have a vendetta, though it certainly seems like it.

Allow me to explain.

I would see a potential “vendetta” driver driving down a street at a relatively constant pace. I am pretty sure they intended to remain at that pace. A jaywalker suddenly comes into the street. The jaywalker has calculated that they can make it across before the car comes upon them. Suddenly, the car speeds up.

Was the driver speeding up by chance alone?

Did the driver just remember that they were late to a baseball game and opted to hit the gas? Or, was it that the driver saw the jaywalker and purposely wanted to give the jaywalker a scare? Suppose you are a driver that has reached your personal threshold with those darned jaywalkers. You might decide that whenever you see one, you will show them who is the boss. You speed up and see how close you can cut it to nearly hitting the jaywalker. In fact, if the jaywalker backs away, you are perhaps just as happy and feel like you did your civic duty.

Here’s something else that seems to go into the jaywalking equation. Does the jaywalker have any kind of encumbrance like a heavy backpack, or maybe a briefcase, or carrying a box or some other object? This tends to slow down the jaywalker and requires them to find a somewhat wider opening in terms of time and space. You might think of this as a kind of golf-like handicap.

Some jaywalkers did not appear to include their encumbrance in their jaywalking formulation. Whereas before they could more readily make a window of X and Y, they know could only do a Q and Z, but they seemed to still be making a jaywalking move when it was only an X and Y window. Danger ensued. Likewise, the drivers would sometimes misjudge the pace and agility of the jaywalker, getting darned close, closer than it seemed they intended, partially because they also miscalculated the delay factor of the encumbrance of the jaywalker.

Nighttime jaywalking was somewhat akin to daylight jaywalking as long as the street was well lit (assuming everyone involved was sober). On some NYC streets, the lighting is not so good. This increased the chances of sour encounters between jaywalkers and drivers. The jaywalker at times seemed to think that the darkness was handy, hiding their jaywalking transgression. The drivers were less likely to see the jaywalkers and get started when the headlights of the car shone upon an unexpected jaywalker.

You can combine all of my aforementioned factors and make the jaywalking into a rather complicated game of human versus human. Human jaywalkers that have human frailties and can misjudge when and how to jaywalk. Weather conditions that can impact the game, along with daylight versus darkness. Drivers that pay attention and other drivers that do not. Some that have a vendetta, some that are drunk.

It’s a real mishmash.

Overall, it is kind of startling and amazing that there aren’t more injuries and deaths due to jaywalking, especially in cities that take jaywalking for granted and it doesn’t get suppressed or expunged.

Of course, not all countries are necessarily opposed to jaywalking. On an international basis, there are some places in the world that jaywalking is strictly forbidden, and other places that allow it and give no special heed about it. You might find of idle interest that the root word “jay” essentially means inexperienced, and when cars first came onto roadways there were drivers that drove on the wrong side of the street, which were referred to as jay-drivers. This morphed eventually into becoming jaywalkers.

Right Of Way As Jaywalking Rules

Who has the proper right of way?

In some countries, both the jaywalker and the driver are considered equals in terms of right-of-way. Once a jaywalker starts across the street, in some countries this implies that the jaywalker now has the true right-of-way, subject to whether the jaywalker has made a sensible move or not. If the jaywalker steps into the street in front of a car going 60 miles per hour and there was no chance for the driver to stop, the jaywalker cannot expect to have claimed the right-of-way.

Here’s what the Department of Motor Vehicles (DMV) rulebook in California states about the act of jaywalking:

“(a) Every pedestrian upon a roadway at any point other than within a marked crosswalk or within an unmarked crosswalk at an intersection shall yield the right-of-way to all vehicles upon the roadway so near as to constitute an immediate hazard.

(b) The provisions of this section shall not relieve the driver of a vehicle from the duty to exercise due care for the safety of any pedestrian upon a roadway.”

You’ll notice that the jaywalker is supposed to yield the right-of-way to cars. Notice further that in spite of that aspect, it does mean that a driver can just run over a jaywalker. The driver must also exercise due care, even if a jaywalker is doing something they aren’t supposed to be doing.

AI Autonomous Cars And Jaywalkers

What does this have to do with AI self-driving driverless autonomous cars?

At the Cybernetic AI Self-Driving Car Institute, we are developing AI software for self-driving cars. One crucial aspect involves the AI being able to contend with jaywalkers.

Allow me to elaborate.

I’d like to clarify and introduce the notion that there are varying levels of AI self-driving cars. The topmost level is considered Level 5. A Level 5 self-driving car is one that is being driven by the AI and there is no human driver involved. For the design of Level 5 self-driving cars, the automakers are even removing the gas pedal, brake pedal, and steering wheel, since those are contraptions used by human drivers. The Level 5 self-driving car is not being driven by a human and nor is there an expectation that a human driver will be present in the self-driving car. It’s all on the shoulders of the AI to drive the car.

For self-driving cars less than a Level 5 or Level 4, there must be a human driver present in the car. The human driver is currently considered the responsible party for the acts of the car. The AI and the human driver are co-sharing the driving task. In spite of this co-sharing, the human is supposed to remain fully immersed into the driving task and be ready at all times to perform the driving task. I’ve repeatedly warned about the dangers of this co-sharing arrangement and predicted it will produce many untoward results.

For my overall framework about AI self-driving cars, see my article: https://aitrends.com/selfdrivingcars/framework-ai-self-driving-driverless-cars-big-picture/

For the levels of self-driving cars, see my article: https://aitrends.com/selfdrivingcars/richter-scale-levels-self-driving-cars/

For why AI Level 5 self-driving cars are like a moonshot, see my article: https://aitrends.com/selfdrivingcars/self-driving-car-mother-ai-projects-moonshot/

For the dangers of co-sharing the driving task, see my article: https://aitrends.com/selfdrivingcars/human-back-up-drivers-for-ai-self-driving-cars/

Let’s focus herein on the true Level 5 self-driving car. Much of the comments apply to the less than Level 5 self-driving cars too, but the fully autonomous AI self-driving car will receive the most attention in this discussion.

Here’s the usual steps involved in the AI driving task:

  • Sensor data collection and interpretation
  • Sensor fusion
  • Virtual world model updating
  • AI action planning
  • Car controls command issuance

Another key aspect of AI self-driving cars is that they will be driving on our roadways in the midst of human driven cars too. There are some pundits of AI self-driving cars that continually refer to a utopian world in which there are only AI self-driving cars on the public roads. Currently there are about 250+ million conventional cars in the United States alone, and those cars are not going to magically disappear or become true Level 5 or Level 4 AI self-driving cars overnight.

Indeed, the use of human driven cars will last for many years, likely many decades, and the advent of AI self-driving cars will occur while there are still human driven cars on the roads. This is a crucial point since this means that the AI of self-driving cars needs to be able to contend with not just other AI self-driving cars, but also contend with human driven cars. It is easy to envision a simplistic and rather unrealistic world in which all AI self-driving cars are politely interacting with each other and being civil about roadway interactions. That’s not what is going to be happening for the foreseeable future. AI self-driving cars and human driven cars will need to be able to cope with each other.

For my article about the grand convergence that has led us to this moment in time, see: https://aitrends.com/selfdrivingcars/grand-convergence-explains-rise-self-driving-cars/

See my article about the ethical dilemmas facing AI self-driving cars: https://aitrends.com/selfdrivingcars/ethically-ambiguous-self-driving-cars/

For potential regulations about AI self-driving cars, see my article: https://aitrends.com/selfdrivingcars/assessing-federal-regulations-self-driving-cars-house-bill-passed/

For my predictions about AI self-driving cars for the 2020s, 2030s, and 2040s, see my article: https://aitrends.com/selfdrivingcars/gen-z-and-the-fate-of-ai-self-driving-cars/

Returning to the topic of jaywalkers, let’s consider the capabilities that an AI self-driving car should have regarding contending with these wayward pedestrians.

I’ll tackle right away a comment that I sometimes get from AI developers.

There are some that say there is no need for an AI self-driving car to do anything at all about a jaywalker.

Jaywalkers are acting illegally.

They get whatever they deserve.

There is no requirement that the AI of the self-driving car needs to do anything at all about a jaywalker.

It might seem astonishing to you that someone would think this way. It is a phenomenon that I refer to as the “egocentric” developer viewpoint. The world needs to conform to their view of the world, rather than the developer facing the reality of the real-world. I quickly point out to such a person that the DMV code clearly states that the car driver must exercise a duty of care, even if the jaywalker is doing something utterly wrong and illegal.

This often surprises the AI developer. They had laid all the responsibility onto the shoulders of the wayward pedestrian. In one sense, this is similar to the jaywalkers that insist they are doing a “victimless” act and that it is up to the jaywalker to choose whether to personally risk going jaywalking or not. Only after I point out the other “victims” that can get dragged into the “victimless” effort do they (hopefully) see the larger picture.

For my article about egocentric AI developers, see: https://stage.aitrends.com/selfdrivingcars/egocentric-design-and-ai-self-driving-cars/

For AI developer burnout impacts, see my article: https://stage.aitrends.com/selfdrivingcars/developer-burnout-and-ai-self-driving-cars/

For my article about pedestrian “roadkill” aspects, see: https://aitrends.com/selfdrivingcars/avoiding-pedestrian-roadkill-self-driving-cars/

For my article about the dangers of groupthink among developers, see: https://stage.aitrends.com/selfdrivingcars/groupthink-dilemmas-for-developing-ai-self-driving-cars/

Needing To Contend With Jaywalkers

Let’s all assume that indeed the AI of the self-driving car does need to contend with jaywalkers.

It cannot ignore them.

It cannot pretend that the burden of safety is solely on the backs of the jaywalkers.

The AI must have provisions for dealing with jaywalkers.

I’d say that’s a prudent and societally expected assumption about AI self-driving cars.

This moves us then into the next kind of quirk that some AI developers offer. There are some AI developers that will concede the notion of doing something about jaywalkers, but then argue that a jaywalker is nothing special and that the “normal” driving aspects of an AI self-driving car should suffice when dealing with jaywalkers.

In this case, the AI developer is suggesting that if the AI self-driving car is already prepared to cope with objects that might appear in the roadway, the job of having the AI be prepared for jaywalkers is already completed. No need to do anything else.

This implies that a jaywalker is no different from say a tumbleweed. If the AI is able to detect a tumbleweed in the roadway, it amounts to the same thing as detecting a human in the roadway. At least that’s the kind of thinking involved by this kind of AI developer.

If I was driving my car and saw a tumbleweed in the road, I would likely mentally calculate whether to hit it or not. I might be willing to hit the tumbleweed due to the aspect that perhaps there are other cars near me and if I hit my brakes suddenly, I risk getting rear-ended, and maybe I cannot switch lanes without endangering a car adjacent to me, and maybe radically swerving is likewise going to endanger me and other nearby cars and pedestrians. So, I might choose to ram the tumbleweed, doing so as the “safest” option available to me at the time and moment that the tumbleweed has appeared.

Here’s an easy question for you, I think, namely do you consider it viable to ram a jaywalker, using the same logic about ramming a tumbleweed?

I’d dare say that you would be willing to take much greater chances to avoid hitting the jaywalker than you would hitting a tumbleweed. As an aside, this raises further the ethical aspects involved in driving a car. Suppose you can avoid the jaywalker but might end-up on the sidewalk and hit a pedestrian standing there – what is the basis for making such a choice, and how do we end-up embodying this kind of decision-making into an AI system of a self-driving car?

Back to the object in the roadway problem, do we want AI self-driving cars that seem to equate hitting a human jaywalker is akin to hitting a tumbleweed?

I don’t believe we do.

Thus, I claim that if an AI system is only detecting “objects” and not trying to also figure out what kind of object is involved, it is insufficient in terms of what we would all hope a true AI self-driving car is going to be able to do. From a systems perspective, please realize that I realize that when the cameras, radar, LIDAR, and other sensors first do their detection, they are only indeed detecting “objects” and thus there is a crucial role of object detection involved. What I am saying is that after the raw sensory detection of an object, it is imperative that the AI system tries to discern what kind of object the object is, such as whether it is a tumbleweed or a human.

That’s why the interplay of the sensory detection and sensor fusion is vital.

When the AI system is trying to piece together the sensor data from multiple sensors, it has an enhanced chance of trying to ferret out what kind of object is being dealt with. This also interplays with the virtual world model. The virtual world model should be tracking the object over time, which will also then aid in trying to ascertain what the object might be. The AI action planning capability needs to be “astute” enough to be able to detect patterns of shapes and movement that pertain to humans and try to differentiate this from other kinds of objects.

I purposely have chosen the tumbleweed example because it is a tricky one to discern from the movements of a human.

For example, you might say that a human should presumably start off the street and proceed into the street. Certainly, a tumbleweed could do the same. A jaywalker once in the street is going to likely be making their way across the street. A tumbleweed might do the same, perhaps the wind is pushing it in that direction.

A jaywalker might make a direct beeline across the street. A tumbleweed could do the same. A jaywalker might weave as they cross the street, and of course a tumbleweed might do the same. By the movement alone, you cannot necessarily say whether the object is a human trying to jaywalk versus a tumbleweed.

You would need to combine a multitude of factors. What is the size and shape of the object? Does it resemble the size and shape of a human? Does it move in a seemingly directed fashion, but if so, can this be differentiated from the possible random movements of an object like a tumbleweed? We also need to consider whether the object might be an animal, which could move across the street in the same overall manner that a jaywalker might or a tumbleweed might.

Guessing whether the object is a jaywalker then opens an entire plethora of other aspects for the AI to consider.

My stories about jaywalkers provide ample indication of the kinds of acts that a jaywalker might do. A car driver that is watching the road would indeed adjust their driving behavior based on the realization that a jaywalker is in the road. You might slow down, you might speed-up, you might honk your horn, you might do all kinds of actions as a driver.

Likewise, the AI of a true AI self-driving car should be doing similar kinds of actions.

For the conspicuity aspects of AI self-driving cars, see my article: https://www.aitrends.com/selfdrivingcars/conspicuity-self-driving-cars-overlooked-crucial-capability/

For defensive driving techniques for AI self-driving cars, see my article: https://www.aitrends.com/selfdrivingcars/art-defensive-driving-key-self-driving-car-success/

For my article about AI dealing with roadway debris, see my article:  https://www.aitrends.com/selfdrivingcars/roadway-debris-cognition-self-driving-cars/

For the head nod problem of AI self-driving cars, see my article: https://www.aitrends.com/selfdrivingcars/head-nod-problem-ai-self-driving-cars/

Overreacting To Jaywalkers

Auto makers and tech firms that are making AI self-driving cars are often dealing with just getting an AI self-driving car to deal with the rudiments of driving, and they would say that the best bet is to have the AI always assume the worst-case scenario. This means that a tumbleweed that might be a human is going to be assumed to be a human, which is considered a safer bet than not making that kind of assumption.

They would also tend toward having the AI take the “super-cautious” approach. I remember being invited to watch an AI self-driving car as it drove down an empty street, and the automaker and tech firm had a stuntman walk out into the street, acting like a jaywalker. The AI was able to detect the jaywalker and came to a nearly immediate halt. Success!

Well, not exactly.

The AI self-driving car came to a halt at about one quarter into the block, and the jaywalker was at the other quarter’s end of the block.

Sure, the AI self-driving car detected the jaywalker at a sizable distance and came to a prompt halt at a sizable distance. It was maybe just somewhat less than a half a football field away from the human when it halted.

Great, no chance of hitting that person.

Does this make sense in the real-world?

Imagine if AI self-driving cars are all coming to a grinding halt when detecting a human in the street when the human is many, many car lengths away. Will this be a viable way for AI self-driving cars to make their way on our streets? Suppose all human drivers did the same. You might argue that we’d be safer, but I wonder about how this would really play out.

For example, if you knew that a human driver would always stop for you, wouldn’t you nearly always choose to jaywalk? The moment you see a car coming down the street, just step into the street, and voila, that car is going to come to a halt. You and others could pretty much paralyze all car traffic. Maybe we would end-up with far less jaywalking injuries and deaths, but what would it also do to our ability to use cars as a means of transportation?

There are already reports of people opting to “prank” today’s AI self-driving cars. If you know that an AI self-driving car will stop or maybe turn when you take some kind of action as a pedestrian or maybe when driving in your own car, it is human nature that we would all likely exploit these behaviors of the AI self-driving cars. Want to get ahead and not be behind one of those slower moving AI self-driving cars, easy enough to arrange by tricking the AI self-driving car into slowing down or halting.

For my article about pranking of AI self-driving cars, see: https://www.aitrends.com/selfdrivingcars/pranking-of-ai-self-driving-cars/

For my article about ethical review boards and AI self-driving cars, see: https://www.aitrends.com/selfdrivingcars/ethics-review-boards-and-ai-self-driving-cars/

For the role of greed in car driving see, my article: https://www.aitrends.com/selfdrivingcars/selfishness-self-driving-cars-ai-greed-good/

For the falsehoods of zero fatalities and the advent of AI self-driving cars, see my article: https://www.aitrends.com/selfdrivingcars/self-driving-cars-zero-fatalities-zero-chance/

Use of Machine Learning And Deep Learning

A true AI self-driving car has to be embodied with the kinds of driving skills that humans use, and particularly so with regard to contending with jaywalkers.

It is insufficient to simply rely upon some kind of overarching object detection and assume that doing so will resolve how to cope with jaywalkers. That’s not what human drivers seem to do. The behavior of human drivers is actually quite more complex, and we need to aim for having AI systems that can perform in a like manner.

The AI system needs to incorporate the multitude of factors that I’ve previously mentioned. Is the suspected jaywalker an adult or a child? Is it one person or more than one person? Is there a coupling between the multiple jaywalkers? Might the jaywalker drop something into the roadway, and if so, what contingencies should be considered? Does the weather increase or decrease the chances of jaywalking and is the street that you are driving on more or less prone to jaywalkers? And so on.

Some are hoping that the use of Machine Learning (ML) and Deep Learning (DL) will come to the aid of trying to cope with jaywalkers. In one sense, yes, it can be helpful to use ML and DL, and by collecting large sets of jaywalking circumstances begin to find patterns to suggest how jaywalkers behave, and therefore then have ready-made solutions in-hand by the AI.

I assure you though that today’s kind of ML and DL is not going to be the silver bullet or magic wand that provides the jaywalking kind of driving aptitude needed for a true AI self-driving car. The jaywalker aspects are far too complex. It is not the same as merely analyzing an image to ferret out whether there is a human in the scene or not. This has to do with behaviors and complex ones.

For the Uber incident and my initial analysis, see my article: https://www.aitrends.com/selfdrivingcars/initial-forensic-analysis/

For my second analysis of the Uber incident, see my article: https://www.aitrends.com/selfdrivingcars/ntsb-releases-initial-report-on-fatal-uber-pedestrian-crash-dr-lance-eliot-seen-as-prescient/

For aspects about Machine Learning and Deep Learning, see my article: https://www.aitrends.com/selfdrivingcars/plasticity-in-deep-learning-dynamic-adaptations-for-ai-self-driving-cars/

For safety and AI self-driving cars, see my article: https://www.aitrends.com/selfdrivingcars/safety-and-ai-self-driving-cars-world-safety-summit-on-autonomous-tech/

For the boundaries of AI, see my article: https://www.aitrends.com/selfdrivingcars/ai-boundaries-and-self-driving-cars-the-driving-controls-debate/

Conclusion

Most seasoned drivers tend to take jaywalkers in stride (pun!), meaning that we human drivers can detect jaywalkers, we can anticipate what they might do, we can adjust our driving aspects accordingly, and most of the time the dance leads to the jaywalker getting safely across the street and the car safety proceeding down the street.

This is nearly an effortless act by a seasoned human driver.

AI self-driving cars are not yet as prepared for handling jaywalkers.

The sad incident of the jaywalker in Arizona being run down by an Uber self-driving car is but one example of how limited today’s AI self-driving cars are in terms of coping with jaywalkers. We need to focus greater attention on the AI capability to specifically deal with jaywalkers and not allow the assumed everyday capabilities of the AI to be able to contend properly with jaywalking.

Why did the jaywalker cross the road?

Answer: To safely get to the other side.

Whether you live in a country or place that condones jaywalking or shuns it, in the real-world jaywalking exists and will continue to exist.

In various parts of California, they have started a new effort to discourage jaywalking.

If a jaywalker is caught jaywalking and it is their first such caught offense, the city will give them a bright colored vest and an LED light for free, and tell them that if they continue to jaywalk, which they are not supposed to do, they should at least wear the vest and hold the LED light up (and no ticket for jaywalking is issued).

You might think this a rather “odd” kind of solution to the problem of jaywalking.

Some think it is ingenious, others think it is outright ludicrous.

In any case, let’s hope that AI developers keep attune to advancing self-driving cars toward ensuring that jaywalkers are well-detected and avoided, which is a crucial matter and not merely an “edge” problem by any means.

Copyright 2020 Dr. Lance Eliot

This content is originally posted on AI Trends.

[Ed. Note: For reader’s interested in Dr. Eliot’s ongoing business analyses about the advent of self-driving cars, see his online Forbes column: https://forbes.com/sites/lanceeliot/]