Tuesday, 30 April 2019

Getting Real with AI in Healthcare: MGH Training Machine Learning Models; US, UK Speeding Oncology Trials

By Deborah Borfitz, Senior Science Writer, Cambridge Healthcare

Thinking about artificial intelligence (AI) has changed dramatically over time and has become about as hard to describe as art, says Jeff Fried, director of product management for data management company InterSystems. Machine learning (ML) is commonly considered a subset of AI—often a necessary component, in fact—but also encompasses methodologies such as logistic regression once referred to simply as statistics.

The data is in any case “much more important than the algorithms, which are pretty much the same as they were 20 years ago,” says Fried. Data scientists spend most of their time choosing, gathering, combining, structuring, and organizing data so an algorithm can generate meaningful patterns.

While the volume of digital healthcare data has grown exponentially, Fried says, accessing it can be a challenge due to ever-present security and privacy concerns and because it’s often locked in siloes—even across departments within the same organization. But barriers have fast become opportunities thanks to greater availability of open-source data from places like the National Library of Medicine, giving ML algorithms more data to chew on, and the FHIR standard developed by HL7 International that enables seamless, on-demand information exchange.

Enabling data interoperability and sharing on a statewide and national basis is also a core capability of InterSystems, he adds, as is capitalizing on the potential of AI in medicine now computing resources are available in the cloud. Computing is also more affordable for those doing it in their own data center.

Pathology Project

InterSystems’ approach to AI is a sensible one: pick the right problem, get an early win, build on that momentum and repeat. A recent project with Massachusetts General Hospital (MGH), to improve the accuracy of genomic data generated by its Center for Integrated Diagnostics (CID), is a case in point, says Fried. It began by having an ML model quietly train in the background to identify potential risk patterns for cancer based on what pathologists had themselves discovered.

The primary output of the genomic sequencer—roughly 400 million discrete readouts of short fragments of DNA per run—defied human interpretation, Fried notes. For a single tumor sample on one patient, about 2,000 variants are detected but generally only about two or three get reported to oncologists and everything else is considered “noise” and of no use in guiding treatment decisions.

As part of the project, InterSystems helped MGH built a data lake that serves as a single source of data from its laboratory information system and houses an open-access ML library used for data exploration, production, and deployment to break down data silos within the CID. Maciej Pacula, team lead for computational pathology at MGH, has compared it to a “Facebook activity feed” used by everyone from lab technicians and attendings to people on the bioinformatics team, Fried says.

ML calculations use the Random Forest algorithm that shows the justification for positive or negative decisions, says Fried. The model makes a highly sensitive call, but also looks at the noise pathologists would have rejected trying to find any diamonds in the rough worthy of additional manual inspection. Overall, pathologists end up spending less time per case with the ML-powered decision support tool, he adds.

The cross-check system for ensuring next-generation sequencing data is tied to the right patient includes verifying that the predicted gender matches what’s in the electronic health record, he continues. Previous genotyped results, when available, also get compared with current results to be sure the overlap between detected alterations is as high as expected.

InterSystems also helped develop an open-access Survival Portal that allows MGH oncologists to look at a cohort of past patients with the same genetic signature as a current patient to determine which drug might deliver the best outcomes, says Fried. The patterns it reveals might support hypothesis generation for future clinical trials.

Next up will be a tool for predicting microsatellite instability in cancer patients, based on mutations in the targeted cancer panel, who can then be treated with Keytruda, Fried says. MGH hopes to turn the tool into a clinical screening test.

MGH plans to replicate what it did in its Center for Integrated Diagnostics in other areas of the organization such as cytometry, says Fried. It is also making a big push into digital pathology and deep learning.  

Holdups in the Healthcare Arena

One reason AI has had a hard time getting out of research realm and into the high-stakes clinical arena is that results are “data-dependent, fuzzy and non-deterministic,” says Fried, plus the technology is changing rapidly. “If you come at this as an IT-oriented project it will drive you crazy. You can’t set a quality threshold and a time goal at the same time.” Even when put into production, some data attributes might change that can affect the model in unpredictable ways, he says.

But AI clearly makes sense in clinical settings to address operational inefficiencies and “turbo charge” bread-and-butter processes, says Fried. For example, InterSystems partner HBI Solutions uses AI to predict both the risk of patient readmission and entry to the emergency room. “Accuracy rates are dependent on the local data and environment, but even at the low end of the accuracy range you can get a lot of benefit.”

Improving hospital workflow is one way that AI tends to be used in clinical settings, Fried says. The other is for decision support in specialty areas like pathology and radiology with lots of data and “smart people who are going to pay attention, so the training sets end up being really good and the risk of error is mitigated by the expert.”

Bias in any ML model is practically a given, he adds. “Outside of healthcare we run into this everywhere. Banks that do risk predictions on loans have well-known biases that end up being racist, and even conscientious scientists end up with very pernicious models. The college admissions process is completely screwed up in part because everyone is gaming their US News & World Report ratings and their very sophisticated ML models use flawed proxies, like someone’s likeliness to contribute to their alma mater after they graduate.”

One of the best antidotes to bias is to put humans in the loop to bring in some common sense, Fried says. Of course, humans have their own biases but, unlike machines, tend to know they do. “Machine learning algorithms can also be over-trained by giving them too many examples of the same thing”—an   ML model built to predict car color in a certain geography might over-predict for the popular color white, he cited as an example.

In pathology, an ML algorithm might be similarly over-trained on old and perhaps abandoned practices. “Don’t be afraid of retraining,” he advises. “In the domain of online merchandising people typically retrain from scratch at least every two weeks because consumer behavior changes a lot, plus there’s no dearth of data or data access problems and the risks of being wrong are low.”

Machine learning might one day be used in scenarios not yet imagined, such as having push cart vendors monitoring hospital air quality or people monetizing their personal health data over the course of the flu, says Fried. He notes that the Port of New Bedford is a leader in an initiative of the National Oceanic and Atmospheric Administration that is equipping fishing vessels with sensors to find fish in exchange for paying fishermen a small stipend to record water temperature on their daily travels.  

Addressing the Clinical Trial Recruitment Challenge

AI’s potential to speed up the patient screening process for oncology trials will soon be put to the test at up to 20 sites in the U.S. and the U.K., according to Andrew Rosner, vice president, real world and late phase for Syneos Health (previously INC Research/inVentiv Health). The first-of-its-kind comparison study (MN-1900) seeks to build upon findings of a pilot study, published earlier this year in Therapeutic Innovation and Regulatory Science, showing the AI-powered prescreening software Mendel Trials found up to 50% more patients than standard practices (i.e., site coordinators manually sifting through charts) at 36,000 times the speed (10 minutes vs. 250 days).

The initial study looked at three oncology trials at a single cancer center, including one non-enrolling trial where both approaches failed to identify suitable patients. MN-1900, like its predecessor, will retrospectively apply the AI system to trials that have completed enrollment efforts. It will also look for any biases inadvertently built into the AI algorithm, inherent to the traditional screening process, or related to how study staff feel about the technology, says Rosner.

As explained in a YouTube video on the Mendel.ai website, the AI-augmented prescreening process is straightforward once Mendel Trials is integrated with an electronic medical record system to make patient records searchable. Users can copy and paste a protocol as-is and the software sifts through all that information—from claims data to scanned pathology reports and doctors’ free-text notes—highlighting evidence that matches the protocol criteria to come up with a list of patients eligible for screening. The search engine can find patients for nearly any protocol criterion, including biomarkers, comorbidities, interventions and lab results.

Syneos Health, a “lab to life” organization supporting biopharma in developing and commercializing therapies, is partnering with Mendel.ai on MN-1900 in the belief that advances in healthcare will come from nontraditional players in need of an insider’s view of the industry, says Rosner. It is “all too common” that trial sponsors either extend their original timelines or increase the number of study sites to meet enrollment goals. AI software like Mendel Trials helps solve these challenges, he says, by helping sites quickly surface eligible subjects and only accept trials where they can successfully meet protocol requirements. “This is just something that was not possible in the past.”

Recruitment for MN-1900 is underway, with Mendel indicating it is currently in advanced discussions with multiple sites and that many more are expressing an interest. Rosner says, “We’d like to be reporting out results at SCOPE [Summit for Clinical Ops Executives] in 2020.”

For more information, go to InterSystems and Syneos Health.

Is There Free-Will in Humans or AI: Useful Debate and for AI Self-Driving Cars Too

By Lance Eliot, the AI Trends Insider

Perhaps one of the oldest questions asked by humans is whether or not there is free-will. It’s up there with questions such as why we all exist, how did we come to exist, and other such lofty and seemingly intractable queries. If you are anticipating that I’m going to tell you definitively herein whether there is free-will or not, I guess you’ll have to keep reading, and the choice you make will determine the answer to your question.

I’ll pause while you ponder my point.

Okay, let’s get back underway.

Well, since you are now presumably reading these words, I gather that you choose to keep reading. Did you make that choice of your own free-will?

We generally associate free-will with the notion that you are able to act on your own, making your own decisions, and that there isn’t any particular constraint on which way you might go. Things get muddy quite quickly when we begin to dig deeper into the matter.

As I dig into this, please be aware that some people get upset about how to explain the existence of or the lack of free-will, typically because they’ve already come to a conclusion about it, and therefore any discussion on the matter gets things pretty heated up. I’m not intending to get the world riled up herein.

As you’ll see in a few moments, my aim is to layout some of the key landscape elements on the topic, and then bring it into a vantage point that will allow for considering other points I’d like to make about AI and AI systems.

Indulge me as I try to get us there.

Free-Will Or Non-Free-Will: That’s The Question

If I were to suggest that the world is being controlled by a third-party that was unseen and undetected, and that we were all participants in a play being undertaken by this third party, it becomes hard to either prove that you have free-will if you claim you do, or prove that you don’t have free-will.

Were you to contend that you do have free-will, you are actually saying so under a false pretense or belief, since I’ve claimed that you are merely part of a play, and you are unaware of the play taking place and aren’t able to discern the director that is guiding the action. On the other hand, there’s no evidentiary means to prove that you are not exercising free-will, since the director is unseen and the play itself is unknown to us, instead it is merely life as it seemingly unfolds.

You can make this into a combo deal by suggesting that the play is only an outline, and you still have some amount of free-will, as exercised within the confines of the play. The problem though with this viewpoint is that someone else might contend that there is either free-will or there is not free-will, and if you are ultimately still under the auspices of the play, you don’t have true free-will. You have a kind of muted or constrained free-will.

For those that believe in the binary sense of free-will, meaning you either have it entirely and without any reservation and no limits, that’s their version of the binary digit one, and anything else is considered a binary digit of a zero. Therefore, they would reject that you have free-will unless and only if it is utterly unfettered. No gray areas, no fuzzy logic allowed.

Set aside for the moment the third-party notion and consider another perspective.

Maybe everything that seems to be happening is already predetermined, as though it was a script and we are simply carrying out the script. We don’t see the script and don’t realize it is a script. We don’t know how the script came to be, which of course goes along with the idea that we cannot see it and don’t realize it exists.

Somehow, we are making decisions and taking actions that have been already decided. It could be that the script is extensive to the nth degree, covering every word you say, every action you take. Or, it could be a script that has preferred lines and preferred actions, yet you still can do some amount of improvisation.

Once again, proving that you are not abiding by the written script is not feasible, because there isn’t proof of the script and nor that you are acting upon it. In essence, it doesn’t seem likely that under this script milieu we can prove you do or do not have free-will.

Another take on the free-will underpinnings relates to cause and effect.

Perhaps everything that we do is like a link in a very long chain, each link connecting to the next. Any decision you make at this moment is actually bound by the decision that was made moments earlier, which is bound to the one before that, and so on, tracing all that you do back to some origin point. After the original origin point occurred, all else was like a set of dominos, each domino cascading down due to the one that came before it that had moments before fallen down.

The past is the past. The future is not necessarily already written. It could be that this moment, right now, consists of one of those dominos, about to fall, and once it falls, the next thing that happens is as a direct and unyielding result of the falling domino that just fell. In this perspective, the future could be unplanned and open ended, though it is entirely dependent on the decisions and actions that came before.

Some would describe this viewpoint of your steps being either laid out or as being inextricably connected as depicting what might be commonly called fate, typically considered something that has been predetermined and you are in a sense merely carrying it out, an inch at a time. The word destiny is used in a somewhat similar sense, though usually suggested as a target point, rather than the steps in-between, such as it is your destiny to become rich and famous, though how you get there is perhaps not predetermined, yet you will indeed get there.

In the philosophy field, a concept known as determinism (not to be confused with the computer science meaning) is used to suggest that we are bound by this cause-and-effect aspect. You can find some wiggle room to suggest that you might still have free-will under determinism, and so there’s a variant known as hard determinism that closes off that loophole and claims that dovetailing with the cause-and-effect there is no such thing as free-will.

Depending upon which philosopher you happen to meet while getting a cup of java at your local coffee bar, they might be a compatibilistic believer, meaning that both determinism and free-will can co-exist, or they might be an incompatibilistic believer, asserting that if there is determinism then there is no such thing as free-will.

Some are worried that if you deny that free-will exists, it implies that perhaps whatever we do is canned anyway, and so it apparently makes no difference to try and think things through, you could presumably act seemingly arbitrarily. In that case, your arbitrariness is not actually arbitrary, and it is only you thinking that it is, when in fact it has nothing to do with randomness and one way or another it was already predetermined for you. Thus, chuck aside all efforts to try and decide what to do, since the decision was already rendered.

This kind of thinking tends to drive people toward a type of fatalism. At times, they can use this logic to opt to transgress against others, shrugging their shoulders and saying that it was not them per se, it was instead whatever non-free-will mechanism that they assert brought it to fruition.

Of course, under a non-free-will viewpoint, maybe those that kept trying to think things through were meant to do so, as due to the third-party or due to the script or due to the cause-and-effect, while people that shift into seemingly being purely arbitrary are actually under the spell of one of those predetermined approaches.

One additional twist is the camp that believes in free-won’t.

Let’s consider the free-won’t aspects.

Maybe you do have some amount of free-will, as per my earlier suggestion that there could be a kind of loosey goosey version, but the manner of how it is exercised involves a veto-like capability.

Here’s how that might work. You non-free-will aims to get you to wave your arm in the air, which accordingly you would undertake to do, since we’re saying for the moment you don’t have free-will to choose otherwise.

The free-won’t viewpoint is that you do have a kind of choice, a veto choice. You could choose to not do the thing that the non-free-will stated, and therefore you might choose to not wave your arm. In this free-won’t camp, note that you weren’t the originator of the arm waving. You were the recipient of the arm waving command, yet you were able to exercise your own discretion and veto the command that was somehow otherwise given to you.

An important construct usually underlying this viewpoint is that you could not choose to do anything else, since that’s up to the non-free-will origination aspects, and all you can do is choose to either do or not do the thing that the non-free-will commanded. Thus, your veto could be to not wave your arm, but you cannot then decide to kick your feet instead. Nope. The kicking of your feet has to originate via the non-free-will, of which then your free-won’t get-out-of-jail card allows you to decide not to kick your feet, if that’s what you want to choose to do.

Those that are the binary types will quickly say you obviously don’t have free-will in the use case of having free-won’t, in that you don’t have true free-will, and you have this measly free-won’t, a far cry from an unburdened free free-will. Others would say that you do have free-will, albeit maybe somewhat limited in scope and range.

I think that lays enough groundwork for moving further into the discussion overall. Do keep in mind that the aforementioned indication is just the tip of the iceberg on the topic of free-will. I’ve left out reams of other angles on the topic. Consult your philosophers stone for further information about free-will.

Can Free-Will Be Detected Via Neuroscience

So far, it’s been suggested that for humans, we really cannot say for sure whether we have free-will or not. You can make a claim that we do have free-will, but you then have to presumably prove that there isn’t this non-free-will that is over-the-top of free-will. Some say that the burden of proof needs to be on the non-free-will believers, meaning they need to showcase proof of the non-free-will, otherwise the default is that there is free-will.

Another means to try and break this logjam might be to find one “provable” instance of the existence of free-will, which at least then you could argue that free-will exists, though maybe not all the time and nor everywhere and nor with everyone.

Likewise, some say that if you could find one “provable” instance that there is the existence of non-free-will, you could argue that there is at least one case of non-free-will that presumably overpowers free-will, which might not be the case all the time or for everywhere and nor for everyone, yet it does nonetheless exist (if so proven).

This fight over free-will has drawn scrutiny by just about every domain or discipline that bears on the topic. The field of philosophy is the most obvious such domain. There is also the field of psychology, trying to unlock the mysteries of the mind, as does the field of cognitive science. We can also pile into this the neurosciences, which likewise aims to gauge how the brain works, and ultimately how the brain arrives at the act of thinking.

One key study in neuroscience that sparked quite a lot of follow-on effort was undertaken by Benjamin Libet, Curtis Gleason, Elwood Wright, and Dennis Pearl in 1983 (see https://academic.oup.com/brain/article-abstract/106/3/623/271932).

In their study, they attempted to detect cerebral activity and per their experiment claimed that there was brain effort that preceded conscious awareness of performing a physical motor-skilled act by the human subjects, as stated by the researchers:

“The recordable cerebral activity (readiness-potential, RP) that precedes a freely voluntary, fully endogenous motor act was directly compared with the reportable time (W) for appearance of the subjective experience of ‘wanting’ or intending to act. The onset of cerebral activity clearly preceded by at least several hundred milliseconds the reported time of conscious intention to act.”

Essentially, if you were told to lift your arm, presumably the conscious areas of the brain would activate and send signals to your arm to make it move, which all seems rather straightforward. This particular research study suggested that there was more to this than meets the eye. Apparently, there is something else that happens first, hidden elsewhere within your brain, and then you begin to perform the conscious activation steps.

You might be intrigued by the conclusion reached by the researchers:

“It is concluded that cerebral initiation of a spontaneous, freely voluntary act can begin unconsciously, that is, before there is any (at least recallable) subjective awareness that a ‘decision” to act has already been initiated cerebrally. This introduces certain constraints on the potentiality for conscious initiation and control of voluntary acts.”

Bottom-line, this study was used by many to suggest that we don’t have free-will. It is claimed that this study shows a scientific basis for the non-free-will basis. Furthermore, the time delay between the alleged subconscious effort and the conscious effort initiation became known as Libet’s W, the amount of time gap between the presumed non-free-will and the exercising of some limited kind of free-will (Libet had stated that there might be a free-won’t related to the free-will portion, involving a conscious veto capability).

Not everyone sees this study in the same light. For some, it is a humongous leap of logic to go from the presumed detection of brain activity prior to other brain activity that one assumes is “conscious” activity, and then decide that the forerunner activity had anything at all to do with either non-free-will or free-will.

Many would contend that there is such a lack of understanding about the operations of the brain that making any kind of conclusion about what is happening would be treading on thin ice. There is also the qualm that these were acts involving motor skills, which are presumably going to take much long, orders of magnitude, for the enactment of, due to the physical movements, while the brain itself is able to perform zillions of mental operations in that same length of time.

Does the alleged “unconscious” related brain activity suggest that there is something afoot here, namely that it perhaps supports the theories about a omnipresent third-party that is maybe controlling the brain, or that the script theory is correct and the brain is retrieving a pre-planted script from within the recesses of your noggin, or maybe the cause-and-effect theory is validated since this shows that the “conscious” act was controlled by the “unconscious” causal effect. And so on.

There have been numerous other related neuroscience studies, typically trying to further expound on this W and either confirm or disconfirm via related kinds of experiments. You can likely find as many opponents as proponents about whether these neuroscience studies show anything substantive about free-will.

For my article about the irreproducibility problem, see: https://www.aitrends.com/selfdrivingcars/irreproducibility-and-ai-self-driving-cars/

For my article about the importance of transparency in research, see: https://www.aitrends.com/selfdrivingcars/algorithmic-transparency-self-driving-cars-call-action/

For aspects of plasticity and the brain, see my article: https://www.aitrends.com/selfdrivingcars/plasticity-in-deep-learning-dynamic-adaptations-for-ai-self-driving-cars/

On the topic of self-awareness, take a look at my article: https://www.aitrends.com/selfdrivingcars/self-awareness-self-driving-cars-know-thyself/

Another qualm some have is that these are usually done as retrodiction-oriented studies, meaning that they involve examining the data after-the-fact and trying to interpret and reach conclusions thereof. Some assert that you would need to try and figure out what the brain is doing while it is actually happening, in the midst of acting, rather than recording a bunch of data and then afterward sifting through it.

For those of you are intrigued by this kind of neuroscience pursuit, you might keep your eye on the work taking place at the Institute for Interdisciplinary Brain and Behavioral Sciences at Chapman University, which has Dr. Uri Maoz as the project leader for a multi-million dollar non-federal research grant that was announced in March 2019 on the topic of conscious control of our decisions and actions as humans, along with Dr. Amir Raz, professor of brain sciences and director. Participants in the effort include Charité Berlin (Germany), Dartmouth, Duke, Florida State University, Harvard, Indiana University Bloomington, NIH, Monash University (Australia), NYU, Sigtuna (Sweden), Tel Aviv University (Israel), University College London (UK), University of Edinburgh (UK), and researchers at UCLA and Yale.

Stepwise Actions and Processes

Some would argue that the brain does not necessarily operate in a stepwise fashion and that it is raft with parallelism. Therefore, trying to lay claim that A happens before B is somewhat chancy, when in fact the odds are that A an B are actually happening at the same time or in some kind of time overlapping manner. It is perhaps more nonlinear than it is linear, and only our desire to simplify how things work involves flattening the brain operations into a step-at-a-time sequential description.

Be that as it may, let’s for the moment go along with the notion of an overarching linear progression, and see where that takes us.

Consider that we have a human that is supposed to move their arm, the end result of the effort involves the arm movement, and presumably to get their arm to move there is some kind of conscious brain activity to make it happen.

We have this:

Conscious effort -> Movement of arm

According to some of the related neuroscience research, those two steps are actually preceded by an additional step, and so I need to include the otherwise hidden or unrealized step into the model we are expanding upon herein.

As such:

Unconscious effort -> Conscious effort -> Movement of arm

Let’s add labels to these, as based on what some believe we can so label:

Unconscious effort (non-free-will) -> Conscious effort (free-will that’s free-won’t) -> Movement of arm

Here’s a bit of a question for you, does the conscious effort realize that there is an unconscious effort (namely the unconscious effort that precedes the conscious effort), or is the conscious effort blissfully unaware about the unconscious effort (which presumably launched the conscious effort)?

You might say that the question relates to the earlier discussion about the knowingness or lack thereof about the non-free-will initiations. I’ve stated that some believe there is an undetectable third-party or a laid-out script or a cause-and-effect, none of which are seemingly knowable to us humans and therefore we can neither prove or disprove that these non-free-will controllers are acting upon us.

Maybe the conscious effort is blind to the unconscious effort, and perhaps is acting as though it is under free-will, yet it is actually not.

Or, one counter viewpoint is that maybe the conscious and unconscious work together, knowingly, and are really one overall brain mechanism and it is a fallacy on our part to try and interpret them as separate and disjointed.

Is the conscious effort a process, of its own, running on its own, or so it assumes, or might the unconscious effort and the conscious effort be running in concert with each other?

For that matter, I suppose we could even ponder whether the unconscious effort is knowingly sparking the conscious effort, which instead maybe the unconscious effort is its own independent process and it has no idea that it causes something else to happen after it acts.

I don’t want to go down this rabbit hole to far, for now, and bring up what seems perhaps to be rather abstract in order to make this discussion paradoxically more concrete.

How can we make this more concrete?

Notice that I’ve referred to the unconscious effort and the conscious effort as each being a process. If we shift this discussion now into a computer-based model of things, we might say that we have two processes, running on a computer, and for which they might involve one process preceding the other, or not, and they might interact with each other, or not.

These are processes happening in real-time.

It could be that either of the two processes knows about the other. Or, it could be that the two processes do not know about each other.

For anyone that designs and develops complex real-time computer-based systems, you have likely dealt with these kinds of circumstances. You have one or more processes, operating in real-time, and some of which will have an impact on the other processes, at times being in front of some other process, at other times taking place after some other process, and all of which might or might not be directly coordinated.

Consider a modern-day car that has a multitude of sensors and is trying to figure out the roadway and how to undertake the driving task.

You could have a process that involves collecting data and interpreting the data from cameras that are on the car. You might have a process that does data collection and interpretation of radar sensors. The process that deals with the cameras and the process that deals with the radar could be separate and distinct, neither one communicates with the other, neither one happens before or necessarily after the other. They operate in parallel.

For my article about processes and the classic Sleeping Barber problem, see: https://www.aitrends.com/selfdrivingcars/sleeping-barber-problem-and-ai-self-driving-cars/

For aspects about cognition timing, see my article: https://www.aitrends.com/selfdrivingcars/cognitive-timing-for-ai-self-driving-cars/

For how processes deal with faultiness, see my article: https://www.aitrends.com/selfdrivingcars/fail-safe-ai-and-self-driving-cars/

For the aspects of processes that are devised to argue with each other, see my article:https://www.aitrends.com/features/ai-arguing-machines-and-ai-self-driving-cars/

AI Free-Will Question and Self-Driving Cars Too

What does this have to do with AI self-driving cars?

At the Cybernetic AI Self-Driving Car Institute, we are developing AI software for self-driving cars. The AI system is quite complex and involves thousands of simultaneously running processes, which is important for purposes of undertaking needed activities in real-time, but also offers potential concerns about safety and inadvertent process-related mishaps.

Allow me to elaborate.

I’d like to first clarify and introduce the notion that there are varying levels of AI self-driving cars. The topmost level is considered Level 5. A Level 5 self-driving car is one that is being driven by the AI and there is no human driver involved. For the design of Level 5 self-driving cars, the auto makers are even removing the gas pedal, brake pedal, and steering wheel, since those are contraptions used by human drivers. The Level 5 self-driving car is not being driven by a human and nor is there an expectation that a human driver will be present in the self-driving car. It’s all on the shoulders of the AI to drive the car.

For self-driving cars less than a Level 5, there must be a human driver present in the car. The human driver is currently considered the responsible party for the acts of the car. The AI and the human driver are co-sharing the driving task. In spite of this co-sharing, the human is supposed to remain fully immersed into the driving task and be ready at all times to perform the driving task. I’ve repeatedly warned about the dangers of this co-sharing arrangement and predicted it will produce many untoward results.

For my overall framework about AI self-driving cars, see my article: https://aitrends.com/selfdrivingcars/framework-ai-self-driving-driverless-cars-big-picture/

For the levels of self-driving cars, see my article: https://aitrends.com/selfdrivingcars/richter-scale-levels-self-driving-cars/

For why AI Level 5 self-driving cars are like a moonshot, see my article: https://aitrends.com/selfdrivingcars/self-driving-car-mother-ai-projects-moonshot/

For the dangers of co-sharing the driving task, see my article: https://aitrends.com/selfdrivingcars/human-back-up-drivers-for-ai-self-driving-cars/

Let’s focus herein on the true Level 5 self-driving car. Much of the comments apply to the less than Level 5 self-driving cars too, but the fully autonomous AI self-driving car will receive the most attention in this discussion.

Here’s the usual steps involved in the AI driving task:

  • Sensor data collection and interpretation
  • Sensor fusion
  • Virtual world model updating
  • AI action planning
  • Car controls command issuance

Another key aspect of AI self-driving cars is that they will be driving on our roadways in the midst of human driven cars too. There are some pundits of AI self-driving cars that continually refer to a utopian world in which there are only AI self-driving cars on the public roads. Currently there are about 250+ million conventional cars in the United States alone, and those cars are not going to magically disappear or become true Level 5 AI self-driving cars overnight.

Indeed, the use of human driven cars will last for many years, likely many decades, and the advent of AI self-driving cars will occur while there are still human driven cars on the roads. This is a crucial point since this means that the AI of self-driving cars needs to be able to contend with not just other AI self-driving cars, but also contend with human driven cars. It is easy to envision a simplistic and rather unrealistic world in which all AI self-driving cars are politely interacting with each other and being civil about roadway interactions. That’s not what is going to be happening for the foreseeable future. AI self-driving cars and human driven cars will need to be able to cope with each other.

For my article about the grand convergence that has led us to this moment in time, see: https://aitrends.com/selfdrivingcars/grand-convergence-explains-rise-self-driving-cars/

See my article about the ethical dilemmas facing AI self-driving cars: https://aitrends.com/selfdrivingcars/ethically-ambiguous-self-driving-cars/

For potential regulations about AI self-driving cars, see my article: https://aitrends.com/selfdrivingcars/assessing-federal-regulations-self-driving-cars-house-bill-passed/

For my predictions about AI self-driving cars for the 2020s, 2030s, and 2040s, see my article: https://aitrends.com/selfdrivingcars/gen-z-and-the-fate-of-ai-self-driving-cars/

Let’s return to the discussion about free-will.

AI Systems With Or Without Free-Will

Can an AI system have free-will?

This is a somewhat hotly debated topic these days. There are some that are worried that we are in the midst of creating AI systems that could become presumably sentient, and as a result, maybe they would have free-will.

You might say, great, welcome to the free-will community, assuming you believe that humans have free-will, and might believe it’s a boon to the free-will population to have AI machine-based free-willers around.

On the other hand, some are suggesting that an AI that has free-will might not tow the line in terms of what we humans want the AI to be or do. It could be that the free-will AI decides it doesn’t like us and using its own free-will opts to wipe us from earth or enslave us. This would certainly seem like a rather disappointing turn of events, namely that we somehow spawned free-will into machines and they turn on us, rather than being grateful or at least respectful of us.

There are all sorts of twists and turns in that debate. If we as humans don’t have free-will, presumably the creation of AI would also not have free-will, since it is being crafted by the non-free-will that forced us or led us to make such AI. Or, you could say that the non-free-will decided that it was time to allow for true free-will and figured that doing so might be wasted on humans, and as a result allowed the humans to make something that does have free-will. On and on this goes around.

I’d like to tackle at least one aspect that I believe seems to be to me be relatively clear cut.

For today’s AI, tossing into it the best that anybody in AI can do right now in terms of Machine Learning and Deep Learning, along with deep Artificial Neural Networks, it would seem like this is really still a Turing Machine in action. I realize this is a kind of proof-by-reduction, in which I am saying that one thing reduces to the equivalent of another, but I think it is fair game.

Would anyone of any reasonable nature be willing to assert and genuinely believe that a Turing Machine can somehow embody or exhibit free-will?

I dare say it just seems over-the-top to think it has or could have free-will. Now, I realize that also takes us into the murky waters of what is free-will. Without getting carried away here and having to go on and on, I would shorten this to say that a Turing Machine has no such spark that we tend to believe is part of human related free-will.

I’m sure that I’ll get emails right away and criticized that I’ve said or implied that we cannot ever have AI that might have free-will (if there is such a thing), which is not at all what I’ve said or implied, I believe. For the kind of computer based systems that we use today, I believe I’m on safe ground about this, but I quite openly say that there are future ways of computing that might well indeed go beyond what we can do today, and whether or not that might have a modicum of free-will, well, who’s to say.

For my article about the singularity, see: https://www.aitrends.com/selfdrivingcars/singularity-and-ai-self-driving-cars/

For the super-intelligence future dangers we might face, see my article: https://www.aitrends.com/selfdrivingcars/super-intelligent-ai-paperclip-maximizer-conundrum-and-ai-self-driving-cars/

For conspiracies about AI systems and the takeover, see my article: https://www.aitrends.com/selfdrivingcars/conspiracy-theories-about-ai-self-driving-cars/

For my article about whether AI is a Frankenstein, see: https://www.aitrends.com/selfdrivingcars/frankenstein-and-ai-self-driving-cars/

For the Turing Test and AI, see my article: https://www.aitrends.com/selfdrivingcars/turing-test-ai-self-driving-cars/

AI Self-Driving Cars and Lessons Based on Free-Will Debate

Let’s assume that we are able to achieve Level 5 self-driving cars. If so, does that mean that AI has become sentient? The answer is not necessarily.

Some might say that the only path to a true Level 5 self-driving car involves having the AI be able to showcase common-sense reasoning. Likewise, the AI would need to have Artificial General Intelligence (AGI). If you start cobbling together those aspects and they are all indeed a necessary condition for the advent of Level 5, one supposes that the nearness to some kind of sentience is perhaps increasing.

It seems like a fairly sound bet that we can reach Level 5 without going quite that far in terms of AI advances. Albeit the AI driving won’t perhaps be the same as human driving, yet it will be sufficient to perform the Level 5 driving task.

I’d like to leverage the earlier discussion herein about processes and relate that aspect to AI self-driving cars. This will give a chance to cover some practical day-to-day ground, rather than the otherwise lofty discussion so far about free-will, which was hopefully interesting, and led us to consider some everyday perfunctory matters too.

Let’s start with a use case that was brought up during a recent event by Tesla that was known as their Autonomy Investor Day and involved a car and a bicycle and how the capabilities of automation might detect such aspects (the Tesla event took place on April 22, 2019 at Tesla HQ and was live-streamed on YouTube).

For my article about common-sense reasoning and AI, see: https://www.aitrends.com/selfdrivingcars/common-sense-reasoning-and-ai-self-driving-cars/

For aspects about AGI, see my article: https://www.aitrends.com/selfdrivingcars/genius-shortage-hampers-solving-ai-unsolved-problems-the-case-of-ai-self-driving-cars/

For idealism and AI pursuits, see my article: https://www.aitrends.com/selfdrivingcars/idealism-and-ai-self-driving-cars/

For the dangers of noble cause corruption and AI aspects, see: https://www.aitrends.com/selfdrivingcars/noble-cause-corruption-and-ai-the-case-of-ai-self-driving-cars/

Use Case of The Bike On Or Off The Car

Suppose you have an AI self-driving car that is scanning the traffic ahead. Turns out that there is a car in front of the self-driving car, and this car has a bike that’s sitting on a bike rack, which is attached to the rear of the car. I’m sure you’ve seen this many times. If you want to take your bicycle someplace, you put a bike rack onto the back of your car, and you then mount the bike onto the bike rack.

The variability of these bike racks and mountings can be somewhat surprising.

There are some bike racks that can hold several bikes at once. Some bike racks can only handle one bike, or maybe squeeze in two, and yet the person mounted say four bikes onto it. I’ve seen some mounted bikes that were not properly placed into the rack and looked as though they might fall out at any moment.

A friend told me that one time she saw a bike come completely off the bike rack, while a car was in-motion, which seems both frightening and fascinating to have seen. Frightening because a bike that becomes a free wheeling (ha, almost said free-will!) object on the roadway, beyond the control of a human bike rider, well, it is a scary proposition for nearby traffic and nearby pedestrians.

Imagine if you were riding your own bike in the bike lane, minding your own business, riding safely, and another bike suddenly flew off the rear of a car and smashed into you. I dare say no one would believe your story.

Suppose you were driving a car and came upon the madcap bike; it creates difficult choices. A small dropped item like a hubcap you might be willing to simply run over, rather than making a radical and potentially dangerous driving maneuver, but a bike is a sturdier and larger object and one that by striking could do a lot of damage to the car and the bike. In a split second, you’d need to decide which was the better choice, avoid the zany bike and is so doing perhaps endanger yourself and other traffic, or ram into the bike, and possibly endangering yourself or other traffic. Neither option is pleasant.

I did see something about a mounted bike that caught my attention one day. The bike was mounted incorrectly and protruded far beyond the rightmost side of the car. This became a dangerous kind of dagger, poking over into the lane to the right of the car. I wondered whether the driver realized what they had done, or whether they were oblivious and had not realized the predicament that had created for all other nearby car traffic.

I watched as several cars approached in the right lane, adjacent to the car with the improperly mounted bike, which was in the left lane. Those cars often seemed to fail to discern the protruding element until the last moment. Car after car would swerve suddenly to their right, attempting to avoid the spoked wheel of the bike. The swerving was not overly dangerous when there was no other traffic to the further right, but when there was other such traffic, the swerving avoiders would cause other cars in those further right lanes to also weave and semi-panic.

In any case, let’s consider that there is a process in the AI system that involves trying to detect cars that are nearby to the AI self-driving car. This is typically done as a result of Machine Learning and Deep Learning, involving a deep Artificial Neural Network getting trained on the images of cars, and then using that trained capability for real-time analyses of the traffic surrounding the self-driving car.

You might have a second process that involves detecting bicycles. Once again, it is likely the process was developed via Machine Learning and Deep Learning and consists of a deep Artificial Neural Network that was trained on images of bikes.

For the moment, assume then that we have two processes, one to find cars in the camera images and video streaming while the self-driving car is underway, and a second process to find bicycles.

For my article about street scene analyses by AI, see: https://www.aitrends.com/selfdrivingcars/street-scene-free-space-detection-self-driving-cars-road-ahead/

For aspects about federated Machine Learning, see my article: https://www.aitrends.com/selfdrivingcars/federated-machine-learning-for-ai-self-driving-cars/

For my article about Deep Learning aspects, see: https://www.aitrends.com/selfdrivingcars/ai-machine-child-deep-learning-the-case-of-ai-self-driving-cars/

For the handling of roadway debris by AI self-driving cars, see my article: https://www.aitrends.com/selfdrivingcars/roadway-debris-cognition-self-driving-cars/

During the Tesla event, an image was shown of a car with a bike mounted on a rear bike rack. It was demonstrated that the neural network automation was detecting both the car and the bike, each as independent objects.

Now, this could be disconcerting in one manner, namely if the AI is under the belief that there is a car ahead of the self-driving car, and there is also a bike ahead of the self-driving car, each of which is doing their own thing. You might be startled to think that these would be conceptually two different matters. As a human, you know that the bike is really mounted on the car and not under its own sense of motion or actions. The bike is going along for the ride, as it were.

I guess you could say that the bike has no free-will at this moment and is under the non-free-will exerted control of the car.

If the AI though is only considering the car as a separate matter, and the bike as a separate matter, it could get itself tied into a bit of a knot. The bike is facing in some particular direction, depending upon how it was mounted, so let’s pretend it is mounted with the handle bars on the right-side of the car. The programming of the AI might be that it assumes a bicycle will tend to move in the direction of the handlebars, normally so.

Imagine the curious nature then of what the AI is perceiving. A car is ahead of the self-driving car. It is moving forward at some speed and distance from the self-driving car. Meanwhile, there’s a bike that is just at the same distance, moving at the same speed but doing so in an oddball manner, it is moving forward yet it is facing to the side.

Where is the bike next going to be?

The standard assumption would be that the bike will be moving to the right, and thus it would be a reasonable prediction to anticipate that the bike will soon end-up to the right. If the car with the mounted bike continues straight ahead, the bike obviously won’t end-up going to the right. Of course, if the car with the mounted bike were to move into the right lane, it would likely lend credence to the notion that the bike is moving and has now been bicycled into the right lane.

One viewpoint of this matter from an AI systems perspective is that the car ahead should be considered as a large blob that just so happens to have this other thing on it, but that it doesn’t care what that thing is. All that is needed is to realize that the car is of a size NxM, which encompasses the added scope of the bike.

So, we have two processes, one finding cars, one finding bikes, and the bike finding process is potentially misleading the rest of the AI system by trying to clamor that there is a bike ahead of the self-driving car. The AI developers realized that this is both true and false at the same time, being that there is a bike there, but it is not a free-wheeling bike.

One reaction by the AI developers involves “fixing” the AI system to ignore a bike when it is seemingly mounted on the back of a car. There is presumably no need to detect such a bike. It doesn’t matter that it so happens to be a bike. If the car had a piano mounted on the back of the car, it wouldn’t matter that it was a piano, and instead merely noteworthy that the car is larger in size than might usually be the case (when you include the scope of the piano).

I certainly grasp this approach, yet it also seems somewhat worrisome.

A human knows that a bike is a bike. A bike has wheels and it can roll around. A human knows that a bike mounted on the back of a car can come loose. A bike that comes loose can possibly fall onto the roadway like a wooden pallet, making a thud and not going anywhere, or it could potentially move more freely due to the wheels. Of course, without a bike rider, presumably the bike is not going be able to ride along per se, yet with the motion already underway as a result of being on the car, there’s a chance that the bike could “roll” for some distance.

You might be objecting and saying that the odds of a bike coming off a bike rack is slim, and it would also seem slim that once the bike did fall off that it would move along on the roadway. As such, with such slim odds, it seems like a rather remote edge case and you can just reduce the whole topic to not caring about the bike, instead relying upon some other part of the AI that might deal with debris that falls onto the street.

The counter argument is that it is still worthwhile to realize that the bike is a bike, being able to therefore gauge what might happen if the bike does fall off the car. It might be best to be proactive and anticipate that such a mishap might occur, rather than waiting until it does happen and having to react, not having gotten prepared for the possibility of the mishap.

This all ties too to the topic of how much should AI systems be doing defensive driving tactics, which most are not yet doing. By-and-large, the focus by most auto makers and tech firms has been the reactive side of driving. React once something happens is the focus, rather than trying to anticipate what might happen. Novice drivers tend to be the same way.

I’ve emphasized many times in my writings and speeches that the lack of defensive driving tactics for the AI systems will make them brittle and vulnerable. I don’t view that defensive driving as an edge or corner case to be handled at some later time, which regrettably some others do.

For more about edge or corner cases, see my article: https://www.aitrends.com/selfdrivingcars/edge-problems-core-true-self-driving-cars-achieving-last-mile/

For my article about the aspects of defensive driving tactics, see: https://www.aitrends.com/selfdrivingcars/art-defensive-driving-key-self-driving-car-success/

For why AI self-driving cars need a bit of greed, see my article: https://www.aitrends.com/selfdrivingcars/selfishness-self-driving-cars-ai-greed-good/

For my article about the brittleness aspects, see: https://www.aitrends.com/ai-insider/machine-learning-ultra-brittleness-and-object-orientation-poses-the-case-of-ai-self-driving-cars/

For the role of micro-movements, see my article: https://www.aitrends.com/ai-insider/micro-movements-in-driving-behaviors-crucial-for-ai-self-driving-cars/

Conclusion

When discussing the topic of free-will, it can become quite abstract and tilt towards the theoretical and the philosophical side of things. Such discussions are worthwhile to have, and I hope that my offering of a taste of it will be of interest to you, perhaps spurring you to look further into the topic.

I’ve tried to also bring some of the topic to a more day-to-day realm. You can think of the free-will and non-free-will discussion as being about control or lack-of-control over processes (in a more pedantic, mundane way, perhaps).

When developing real-time AI systems, such as AI self-driving autonomous cars, you need to be clearly aware of how those processes are running and what kind of control they have, or lack thereof.

If you are the type of reader that began reading this article and upon my opening remark that maybe or maybe not that I would reveal whether humans have free-will, and if you then skipped the entire piece and jumped just to this conclusion, in hopes of seeing what I proclaimed, well, you’ll have to do the hard work and actually read the whole piece.

You can then decide whether or not I did state whether free-will exists or not, doing so by your own choice of opting to actually read the piece. That’s of your own free-will. Or is it?

Copyright 2019 Dr. Lance Eliot

This content is originally posted on AI Trends.

 

As Governments Adopt AI, Little Oversight and Lots of Danger

By James Hendler, Rensselaer Polytechnic Institute

Artificial intelligence systems can – if properly used – help make government more effective and responsive, improving the lives of citizens. Improperly used, however, the dystopian visions of George Orwell’s “1984” become more realistic.

On their own and urged by a new presidential executive order, governments across the U.S., including state and federal agencies, are exploring ways to use AI technologies.

As an AI researcher for more than 40 years, who has been a consultant or participant in many government projects, I believe it’s worth noting that sometimes they’ve done it well – and other times not quite so well. The potential harms and benefits are significant.

In an early success, the U.S. Department of Homeland Security in 2015 developed an AI system called “Emma,” a chatbot that can answer questions posed to it in regular English, without needing to know what “her” introductory website calls “government speak” – all the official terms and acronyms used in agency documents.

By late 2016, DHS reported that Emma was already helping to answer nearly a half-million questions per month, allowing DHS to handle many more inquiries than it had previously, and letting human employees spend more time helping people with more complicated queries that are beyond Emma’s abilities. This sort of conversation-automating artificial intelligence has now been used by other government agencies, in cities and countries around the world.

Experience with Flint, Mich. Water Supply

A more complicated example of how governments could aptly apply AI can be seen in Flint, Michigan. As the local and state governments struggled to combat lead contamination in the city’s drinking water, it became clear that they would need to replace the city’s remaining lead water pipes. However, the city’s records were incomplete, and it was going to be extremely expensive to dig up all the city’s pipes to see if they were lead or copper.

Instead, computer scientists and government employees collaborated to analyze a wide range of data about each of 55,000 properties in the city, including how old the home was, to calculate the likelihood it was served by lead pipes. Before the system was used, 80% of the pipes dug up needed to be replaced, which meant 20% of the time, money and effort was being wasted on pipes that didn’t need replacing.

The AI system helped engineers focus on high-risk properties, identifying a set of properties most likely to need pipe replacements. When city inspectors visited to verify the situation, the algorithm was right 70% of the time. That promised to save enormous amounts of money and speed up the pipe replacement process.

However, local politics got in the way. Many members of the public didn’t understand why the system was identifying the homes it did, and objected, saying the AI method was unfairly ignoring their homes. After city officials stopped using the algorithm, only 15% of the pipes dug up were lead. That made the replacement project slower and more costly.

Many Do Not Trust AI

The problem in Flint was that people didn’t understand that AI technology was being used well, and that people were verifying its findings with independent inspections. In part, this was because they didn’t trust AI – and in some cases there is good reason for that.

In 2017, I was among a group of more than four dozen AI researchers who sent a letter to the acting secretary of the U.S. Department of Homeland Security. We expressed concerns about a proposal to use automated systems to determine whether a person seeking asylum in the U.S. would become a “positively contributing member of society” or was more likely to be a terrorist threat.

“Simply put,” our letter said, “no computational methods can provide reliable or objective assessments of the traits that [DHS] seeks to measure.” We explained that machine learning is susceptible to a problem called “data skew,” in which the system’s ability to predict a characteristic depends in part on how common that characteristic is in the data used to train the system.

So in a database of 300 million Americans, if one in 100 people are, say, of Indian descent, the system will be fairly accurate at identifying them. But if looking at a characteristic shared by just one in a million Americans, there really isn’t enough data for the algorithm to make a good analysis.

As the letter explained, “on the scale of the American population and immigration rates, criminal acts are relatively rare, and terrorist acts are extremely rare.” Algorithmic analysis is extremely unlikely to identify potential terrorists. Fortunately, our arguments proved convincing. In May 2018, DHS announced it would not use a machine learning algorithm in this way.

See the source article in governmenttechnology.com.

Walmart Experiments With an AI-powered Store Open to the Public in Levittown, NY

Walmart recently unveiled a new “store of the future” test ground for emerging technologies, including AI-enabled cameras and interactive displays. The store, a working concept called the Intelligent Retail Lab — or “IRL” for short — operates out of a Walmart Neighborhood Market in Levittown, N.Y.

The store is open to customers and is one of Walmart’s busiest Neighborhood Market stores, containing more than 30,000 items, the retailer says, which allows it to test out technology in a real-world environment.

Similar to Amazon Go’s convenience stores, the store has a suite of cameras mounted in the ceiling. But unlike Amazon Go, which is a grab-and-go store with smaller square footage, Walmart’s IRL spans 50,000 square feet of retail space and is staffed by more than 100 employees.

Plus, in Walmart’s case, these AI-powered cameras are not being used to determine which items customers are buying in order to automatically charge them. It still has traditional checkout stations. Instead, the cameras will monitor inventory levels to determine, for example, if staff needs to bring out more meat from the back-room refrigerators to restock the shelves, or if some fresh items have been sitting too long on the shelf and need to be pulled.

The idea is that the AI will help the store associates know more precisely where and when to restock products. And this, in turn, means customers will know the produce and meat is always fresh and in stock when they arrive.

Using technology to do this is not simple, Walmart says. It means the automated system will need to be able to detect products on the shelf, recognize the exact product it sees (1 lb of ground beef versus 2 lbs., etc.), and then compare the quantities on the shelf to upcoming sales demand.

For store associates, the system allows them to stop constantly walking the store to replace inventory — instead, they’ll know what to bring out from the back room before the doors even open to customers that day.

The cameras and other sensors in the store pump out 1.6 TB of data per second, or the equivalent of three years’ worth of music, which necessitates a big data center on site. At the IRL store, it’s glass-encased, bathed in blue light and on display to the public.

This could seem a little intimidating — AI cameras and giant servers. But Walmart says the data is only stored for less than a week.

There are also informational stations in the store where customers can learn more about the technology in use. A Welcome Center in the store is available too, for customers who want to learn more about the technical specifications and get answers to common questions.

An interactive wall lets customers have fun with AI — it demonstrates how an AI system can estimate body positioning. But really it’s meant to make all this new technology seem less intimidating.

Read the source article at TechCrunch.

Using Dynamic Time Warping and MLflow to Detect Sales Trends

Try this notebook series in Databricks

This blog is part 2 of our two-part series Using Dynamic Time Warping and MLflow to Detect Sales Trends


The phrase “dynamic time warping,” at first read, might evoke images of Marty McFly driving his DeLorean at 88 MPH in the Back to the Future series. Alas, dynamic time warping does not involve time travel; instead, it’s a technique used to dynamically compare time series data when the time indices between comparison data points do not sync up perfectly.

As we’ll explore below, one of the most salient uses of dynamic time warping is in speech recognition – determining whether one phrase matches another, even if it the phrase is spoken faster or slower than its comparison. You can imagine that this comes in handy to identify the “wake words” used to activate your Google Home or Amazon Alexa device – even if your speech is slow because you haven’t yet had your daily cup(s) of coffee.

Dynamic time warping is a useful, powerful technique that can be applied across many different domains. Once you understand the concept of dynamic time warping, it’s easy to see examples of its applications in daily life, and its exciting future applications. Consider the following uses:

  • Financial markets – comparing stock trading data over similar time frames, even if they do not match up perfectly. For example, comparing monthly trading data for February (28 days) and March (31 days).
  • Wearable fitness trackers – more accurately calculating a walker’s speed and the number of steps, even if their speed varied over time.
  • Route calculation – calculating more accurate information about a driver’s ETA, if we know something about their driving habits (for example, they drive quickly on straightaways but take more time than average to make left turns).

Data scientists, data analysts, and anyone working with time series data should become familiar with this technique, given that perfectly aligned time-series comparison data can be as rare to see in the wild as perfectly “tidy” data.

In this blog series, we will explore:

  • The basic principles of dynamic time warping
  • Running dynamic time warping on sample audio data
  • Running dynamic time warping on sample sales data using MLflow

For more background on dynamic time warping, refer to the previous post Understanding Dynamic Time Warping.

Background

Imagine that you own a company that creates 3D printed products. Last year, you knew that drone propellers were showing very consistent demand, so you produced and sold those, and the year before you sold phone cases. The new year is arriving very soon, and you’re sitting down with your manufacturing team to figure out what your company should produce for next year. Buying the 3D printers for your warehouse put you deep into debt, so you have to make sure that your printers are running at or near 100% capacity at all times in order to make the payments on them.

Since you’re a wise CEO, you know that your production capacity over the next year will ebb and flow – there will be some weeks when your production capacity is higher than others. For example, your capacity might be higher during the summer (when you hire seasonal workers), and lower during the 3rd week of every month (because of issues with the 3D printer filament supply chain). Take a look at the chart below to see your company’s production capacity estimate:

Your job is to choose a product for which weekly demand meets your production capacity as closely as possible. You’re looking over a catalog of products which includes last year’s sales numbers for each product, and you think this year’s sales will be similar.

If you choose a product with weekly demand that exceeds your production capacity, then you’ll have to cancel customer orders, which isn’t good for business. On the other hand, if you choose a product without enough weekly demand, you won’t be able to keep your printers running at full capacity and may fail to make the debt payments.

Dynamic time warping comes into play here because sometimes supply and demand for the product you choose will be slightly out of sync. There will be some weeks when you simply don’t have enough capacity to meet all of your demand, but as long as you’re very close and you can make up for it by producing more products in the week or two before or after, your customers won’t mind. If we limited ourselves to comparing the sales data with our production capacity using Euclidean Matching, we might choose a product that didn’t account for this, and leave money on the table. Instead, we’ll use dynamic time warping to choose the product that’s right for your company this year.

Load the product sales data set

We will use the weekly sales transaction data set found in the UCI Dataset Repository to perform our sales-based time series analysis. (Source Attribution: James Tan, jamestansc ‘@’ suss.edu.sg, Singapore University of Social Sciences)


import pandas as pd

# Use Pandas to read this data
sales_pdf = pd.read_csv(sales_dbfspath, header='infer')

# Review data
display(spark.createDataFrame(sales_pdf))

Each product is represented by a row, and each week in the year is represented by a column. Values represent the number of units of each product sold per week. There are 811 products in the data set.

Calculate distance to optimal time series by product code


# Calculate distance via dynamic time warping between product code and optimal time series
import numpy as np
import _ucrdtw

def get_keyed_values(s):
    return(s[0], s[1:])

def compute_distance(row):
    return(row[0], _ucrdtw.ucrdtw(list(row[1][0:52]), list(optimal_pattern), 0.05, True)[1])

ts_values = pd.DataFrame(np.apply_along_axis(get_keyed_values, 1, sales_pdf.values))
distances = pd.DataFrame(np.apply_along_axis(compute_distance, 1, ts_values.values))
distances.columns = ['pcode', 'dtw_dist']

Using the calculated dynamic time warping ‘distances’ column, we can view the distribution of DTW distances in a histogram.

From there, we can identify the product codes closest to the optimal sales trend (i.e., those that have the smallest calculated DTW distance). Since we’re using Databricks, we can easily make this selection using a SQL query. Let’s display those that are closest.


%sql
-- Top 10 product codes closest to the optimal sales trend
select pcode, cast(dtw_dist as float) as dtw_dist from distances order by cast(dtw_dist as float) limit 10

After running this query, along with the corresponding query for the product codes that are furthest from the optimal sales trend, we were able to identify the 2 products that are closest and furthest from the trend. Let’s plot both of those products and see how they differ.

As you can see, Product #675 (shown in the orange triangles) represents the best match to the optimal sales trend, although the absolute weekly sales are lower than we’d like (we’ll remedy that later). This result makes sense since we’d expect the product with the closest DTW distance to have peaks and valleys that somewhat mirror the metric we’re comparing it to. (Of course, the exact time index for the product would vary on a week-by-week basis due to dynamic time warping). Conversely, Product #716 (shown in the green stars) is the product with the worst match, showing almost no variability.

Finding the optimal product: Small DTW distance and similar absolute sales numbers

Now that we’ve developed a list of products that are closest to our factory’s projected output (our “optimal sales trend”), we can filter them down to those that have small DTW distances as well as similar absolute sales numbers. One good candidate would be Product #202, which has a DTW distance of 6.86 versus the population median distance of 7.89 and tracks our optimal trend very closely.


# Review P202 weekly sales  
y_p202 = sales_pdf[sales_pdf['Product_Code'] == 'P202'].values[0][1:53]

Using MLflow to track best and worst products, along with artifacts

MLflow is an open source platform for managing the machine learning lifecycle, including experimentation, reproducibility, and deployment. Databricks notebooks offer a fully integrated MLflow environment, allowing you to create experiments, log parameters and metrics, and save results. For more information about getting started with MLflow, take a look at the excellent documentation.

MLflow’s design is centered around the ability to log all of the inputs and outputs of each experiment we do in a systematic, reproducible way. On every pass through the data, known as a “Run,” we’re able to log our experiment’s:

  • Parameters – the inputs to our model.
  • Metrics – the output of our model, or measures of our model’s success.
  • Artifacts – any files created by our model – for example, PNG plots or CSV data output.
  • Models – the model itself, which we can later reload and use to serve predictions.

In our case, we can use it to run the dynamic time warping algorithm several times over our data while changing the “stretch factor,” the maximum amount of warp that can be applied to our time series data. To initiate an MLflow experiment, and allow for easy logging using mlflow.log_param(), mlflow.log_metric(),  mlflow.log_artifact(), and mlflow.log_model(), we wrap our main function using:


with mlflow.start_run() as run:
    ...

as shown in the abbreviated code below.


import mlflow

def run_DTW(ts_stretch_factor):
    # calculate DTW distance and Z-score for each product
    with mlflow.start_run() as run:
        
        # Log Model using Custom Flavor
        dtw_model = {'stretch_factor' : float(ts_stretch_factor), 'pattern' : optimal_pattern}       
        mlflow_custom_flavor.log_model(dtw_model, artifact_path="model")

        # Log our stretch factor parameter to MLflow
        mlflow.log_param("stretch_factor", ts_stretch_factor)

        # Log the median DTW distance for this run
        mlflow.log_metric("Median Distance", distance_median)

        # Log artifacts - CSV file and PNG plot - to MLflow
        mlflow.log_artifact('zscore_outliers_' + str(ts_stretch_factor) + '.csv')
        mlflow.log_artifact('DTW_dist_histogram.png')

    return run.info

stretch_factors_to_test = [0.0, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5]
for n in stretch_factors_to_test:
    run_DTW(n)

With each run through the data, we’ve created a log of the “stretch factor” parameter being used, and a log of products we classified as being outliers based upon the Z-score of the DTW distance metric. We were even able to save an artifact (file) of a histogram of the DTW distances. These experimental runs are saved locally on Databricks and remain accessible in the future if you decide to view the results of your experiment at a later date.

Now that MLflow has saved the logs of each experiment, we can go back through and examine the results. From your Databricks notebook, select the  icon in the upper right-hand corner to view and compare the results of each of our runs.

Not surprisingly, as we increase our “stretch factor,” our distance metric decreases. Intuitively, this makes sense: as we give the algorithm more flexibility to warp the time indices forward or backward, it will find a closer fit for the data. In essence, we’ve traded some bias for variance.

Logging Models in MLflow

MLflow has the ability to not only log experiment parameters, metrics, and artifacts (like plots or CSV files), but also to log machine learning models. An MLflow Model is simply a folder that is structured to conform to a consistent API, ensuring compatibility with other MLflow tools and features. This interoperability is very powerful, allowing any Python model to be rapidly deployed to many different types of production environments.

MLflow comes pre-loaded with a number of common model “flavors” for many of the most popular machine learning libraries, including scikit-learn, Spark MLlib, PyTorch, TensorFlow, and others. These model flavors make it trivial to log and reload models after they are initially constructed, as demonstrated in this blog post. For example, when using MLflow with scikit-learn, logging a model is as easy as running the following code from within an experiment:


mlflow.sklearn.log_model(model=sk_model, artifact_path="sk_model_path")

MLflow also offers a “Python function” flavor, which allows you to save any model from a third-party library (such as XGBoost, or spaCy), or even a simple Python function itself, as an MLflow model. Models created using the Python function flavor live within the same ecosystem and are able to interact with other MLflow tools through the Inference API. Although it’s impossible to plan for every use case, the Python function model flavor was designed to be as universal and flexible as possible. It allows for custom processing and logic evaluation, which can come in handy for ETL applications. Even as more “official” Model flavors come online, the generic Python function flavor will still serve as an important “catch all,” providing a bridge between Python code of any kind and MLflow’s robust tracking toolkit.

Logging a Model using the Python function flavor is a straightforward process. Any model or function can be saved as a Model, with one requirement: it must take in a pandas Dataframe as input, and return a DataFrame or NumPy array. Once that requirement is met, saving your function as an MLflow Model involves defining a Python class that inherits from PythonModel, and overriding the .predict() method with your custom function, as described here.

Loading a logged model from one of our runs

Now that we’ve run through our data with several different stretch factors, the natural next step is to examine our results and look for a model that did particularly well according to the metrics that we’ve logged. MLflow makes it easy to then reload a logged model, and use it to make predictions on new data, using the following instructions:

  1. Click on the link for the run you’d like to load our model from.
  2. Copy the ‘Run ID’.
  3. Make note of the name of the folder the model is stored in. In our case, it’s simply named “model.”
  4. Enter the model folder name and Run ID as shown below:

import custom_flavor as mlflow_custom_flavor

loaded_model = mlflow_custom_flavor.load_model(artifact_path='model', run_id='e26961b25c4d4402a9a5a7a679fc8052')

To show that our model is working as intended, we can now load the model and use it to measure DTW distances on two new products that we’ve created within the variable new_sales_units:


# use the model to evaluate new products found in ‘new_sales_units’
output = loaded_model.predict(new_sales_units)
print(output)

Next steps

As you can see, our MLflow Model is predicting new and unseen values with ease. And since it conforms to the Inference API, we can deploy our model on any serving platform (such as Microsoft Azure ML, or Amazon Sagemaker), deploy it as a local REST API endpoint, or create a user-defined function (UDF) that can easily be used with Spark-SQL.    In closing, we demonstrated how we can use dynamic time warping to predict sales trends using the Databricks Unified Analytics Platform.  Try out the Using Dynamic Time Warping and MLflow to Predict Sales Trends notebook with Databricks Runtime for Machine Learning today.

 

--

Try Databricks for free. Get started today.

The post Using Dynamic Time Warping and MLflow to Detect Sales Trends appeared first on Databricks.