Friday, 1 May 2020
How AI steered doctors toward a possible Coronavirus treatment
Garner the right technology platform for Crisis Management
Read More on Datafloq
Precision Scheduling of Autonomous and Human-Based Ridesharing (PSAHBR)
By Lance Eliot, the AI Trends Insider
Norfolk Southern Corp is doing a makeover of some rather convoluted trainyards.
Turns out there are freight railroads that funnel into hubs that have been run the same way for over a century. Generally, freight trains roll into these hectic hubs, the workhorse trains sit around idly waiting for their cargo, and when things seem to reach a suitable readiness of loaded trains ready to roll, the freight trains then head-out on their treks.
It is reportedly a remarkably ad hoc activity and overseen by a seat-of-the-pants approach.
The hope by several of the major train firms, acting as freight haulers, will be to transform this seemingly chaotic hub activity into a precision of scheduling and efficiency.
By revamping the freight train operations, there are intentions to make this complicated dance into one that is tightly woven with specific entry and exit times, predicted in-advance, and carefully tracked schedules. Presumably, this will allow for more freight movement, more timely freight movement, and make better use of the railroad’s scarce resources. Think of an airport with the daily and moment-to-moment ballet of planes arriving and departing, doing so based on published schedules, along with sticking to the timetables as much as possible.
Precision Scheduling Railroading (PSR)
The notion of transforming the freight train operations is being referred to as Precision Scheduling Railroading (PSR).
In theory, the PSR approach should be able to achieve the desired boosts in efficiency and effectiveness.
Having done quite a number of business process revamps in my working career, I can attest that the theory is often easier than the practical reality. I’m sure there is a chance that the PSR might at first fail to adequately model the realities of the freight train operations, perhaps leading to worse chaos and poorer efficiencies and effectiveness at the get-go.
It takes a lot of elbow grease to make sure that formerly by-hand efforts are not forsaken as somehow backward and inappropriate. The odds are that those manual methods evolved over many years and include lots of workarounds that keep the trains rolling. It might not be the most efficient approach, but it gets the job done. There is a chance that a new system could upend that approach and inadvertently foul things up, albeit only initially, once the kinks get ironed out.
This effort also needs to consider the ramifications of upstream and downstream vital touch-points.
Will the freight train customers be able to accommodate a more measured schedule?
Those customers are likely making use of processes and operations that assume the hub has an ad hoc schedule. When the hub changes to a more precise and tenacious schedule, those customers will need to likewise alter how they do their business.
I mention this aspect because sometimes a business process change is narrowly focused and fails to consider the cascading impacts. You might fix the hub, but meanwhile all the feeders into the hub and the feeds out of the hub are entering into a set of processes and systems that won’t know what to do with the revamped approach. This will undercut the hub changes and possibly befuddle those that had assumed they would right away witness crucial improvements in efficiencies and effectiveness.
The hub itself has its own constraints too, that need to be considered.
Let’s assume that there are a set number of tracks, N, and you come up with a schedule that assumes there are N+1 tracks, well, that’s going to be a problem for those workers at the hub (not enough tracks to abide by the system produced schedule!).
Or, maybe the scheduling system assumes that all N tracks can be used, but it could be that on some days a given track has problems and needs to be repaired before it can be used, so there are really only N-1 or N-2, etc. available tracks. Does the scheduling system take that kind of contingency into account?
You are going to have some number of trains coming into the hub, a number T. Meanwhile, there are some number of trains trying to exit from the hub, a number X. Can those T coming into the hub do so on that number of tracks N, while at the same time dealing with the X number of trains aiming to get out of the hub on those same tracks N?
In today’s modern age, there are lots of excellent scheduling systems that are used for a variety of industries, and can handle these kinds of complexities, thus the train hub is not unique or an impossible operation to come under PSR. The point is that it is not as easy as it might seem at first glance. Switching over from an older set of processes to a new set can be tricky and slamming in a fancy scheduling system is not something you can do overnight.
I’d like to focus your attention on another kind of scheduling problem that will soon become a notable and visible concern.
It has to do with the advent of ridesharing.
Ridesharing as a Scheduling Problem
A few months ago, I was visiting a company on the East Coast that was outside a downtown area and not especially close to any kind of public transportation. I had been invited to attend a session at the firm that took place each Tuesday and involved a gathering of managers from throughout the firm, all coming to HQ from a variety of places across the United States.
There was a train station that I had been advised would be a handy place for me to arrive at and then use a taxi or ridesharing to get over to the HQ building.
I did so.
After a full-day of discussions, the Tuesday event ended promptly at 6:30 p.m. The firm prided itself on doing things on-schedule and had made sure that each of the sessions started and ended precisely on-time.
I happened to glance out the window of the HQ at 6:15 p.m. and noticed that cars seemed to be gathering out on the nearby streets.
Lots of cars.
It almost looked like a flock of birds that were coming to get some leftover food scraps. There was a lot of hustling and bustling going on. Some cars were cruising back-and-forth, while others were standing still at curbs, and a few had pulled into actual parking spots.
What was happening?
It turns out that the local ridesharing and taxi services all knew that the HQ had these Tuesday events and that the events ended at precisely 6:30 p.m.
As a result, these car services were all vying to be nearby when the exodus of visitors wanted to all head-out.
I rode in one of the ridesharing cars and spoke to the driver. He explained that when the Tuesday meetings first began, only a few of the ridesharing drivers knew about it. They were caught somewhat off-guard and there weren’t enough cars arriving in time for the 6:30 p.m. exit, meaning that many of those that wanted to get a ride had to wait. Furthermore, some drivers arrived belatedly, getting there at say 7:00 p.m., and the rush of people needing rides had by then dissipated.
Word spread among the ridesharing and taxi services that on Tuesday nights at 6:30 p.m. there would be a swelling of demand for rides at this location. Many of these drivers would normally not have much traffic at that time because the area was outside of a downtown location. This HQ event represented a high potential for fares, including some rides that would be longer and more profitable than doing the usual neighborhood and grocery store kinds of runs.
It was fascinating to witness this somewhat “spontaneous” assembly of ridesharing and taxi services to meet the Tuesday night demand. I could see that not all of the assembled cars were going to get riders. There was no predetermined balancing of supply and demand. The driver of my ridesharing lift told me that the Tuesday night occasions had become overburdened with too many lift cars, making the situation into a cutthroat effort to grab riders.
He explained that the closer that a car could physically get to the HQ building, the higher the odds of getting a rider. But most of the other drivers figured this out too, and so they jockeyed to get close to the building. It became a raw act of trying to outmaneuver other cars and push or shove your way closest to the HQ office.
Some wised up and realize that in a first-to-arrive mode, those ridesharing cars and taxis that got there the soonest were able to secure a spot closest to the building. Gradually, the cars began to arrive sooner and sooner, because of the competitiveness of wanting to get one of those vaunted slots nearest to the building. Eventually, some of the ridesharing and taxi cars were arriving a full hour early, simply to get a vaunted spot next to the exit doors of HQ.
You must have some sympathy for these working stiffs, since my driver pointed out to me that by arriving sooner, you did get a higher chance of getting a fare, but this also meant that you likely were spurning other possible fares that you might have gotten at say 5:30 p.m. or 6:00 p.m., since it would have kept you away from getting to the building early.
Which was better, these drivers must have pondered, not be at the building early enough and therefore possibly get other fares elsewhere and but end-up at the low-end of fare chances at 6:30 p.m. or arrive early to the HQ to essentially guarantee you’d get a fare, and yet be idle and unpaid during that waiting time.
A few weeks later, I came back to one of the Tuesday events and found out that the HQ had decided to step into the car lifts aspects and try to straighten things out.
The HQ had made a deal with one particular ridesharing firm for picking up of the riders at 6:30 p.m., doing so by negotiating a special rate for the riders and turning the otherwise “catch as catch can” into a more rigorous process. This meant that the other ridesharing firms and taxi services now realized that at best they might get some leftover crumbs, and so they opted to no longer come over to the HQ to try to get riders.
Interestingly, the problem now for some riders was that there weren’t enough cars available to satisfy the demand. My guess was that the HQ would be talking with the ridesharing firm about ensuring that enough cars would show-up to meet the demand. The lower price of the fares was handy, yet there was also the need to make sure there were enough cars to provide lifts, and not let the wait time get out-of-hand.
This was especially the case since the Tuesday visitors were used to the idea that there would be an overwhelming number of cars and the odds of instantly getting a lift had been extremely high, prior to the switchover to a specific ridesharing firm. The old way of doing things seemed to have led to a tremendous amount of supply of cars and the riders had the upper hand. Now, in this more reasoned approach, the riders seemed to be less catered to. I’m sure that the next time I go to the Tuesday event, those kinks will have been likely worked out.
I hope you can see that this story is yet another example of a type of scheduling problem.
Similar to the freight trains, there are a multitude of needs for transport in this ridesharing example, and a need to figure out the balance of supply and demand.
You might be able to leave things to a Darwinian approach of letting nature kind of work things out, akin to what happened at first with the ridesharing and taxi firms that wanted to serve the riders at HQ, and what has seemingly occurred at the freight train hubs.
Unfortunately, the ad hoc method can be a hit-or-miss and instead, presumably, a well-design and well-implemented approach is likely to produce better results, once it has been put in place and tweaked accordingly.
For my article about ridesharing, see: https://www.aitrends.com/selfdrivingcars/ridesharing-services-and-ai-self-driving-cars-notably-uber-in-or-uber-out/
For my Top 10 predictions about AI self-driving cars, see: https://ift.tt/2VSEgdq
For human in-the-loop aspects, see my article: https://www.aitrends.com/ai-insider/human-in-the-loop-vs-out-of-the-loop-in-ai-systems-the-case-of-ai-self-driving-cars/
For my article about vehicle caravans, see: https://www.aitrends.com/selfdrivingcars/traveling-in-vehicle-caravans-and-the-advent-of-ai-self-driving-cars/
AI Autonomous Cars and Ridesharing Aspects
What does this have to do with AI self-driving driverless autonomous cars?
At the Cybernetic AI Self-Driving Car Institute, we are developing AI software for self-driving cars. Most pundits predict that AI self-driving cars will be used as ridesharing cars, doing so to recoup their cost and earn some added dough by fully utilizing the self-driving cars. This will likely lead to some hefty scheduling issues.
Allow me to elaborate.
I’d like to first clarify and introduce the notion that there are varying levels of AI self-driving cars. The topmost level is considered Level 5. A Level 5 self-driving car is one that is being driven by the AI and there is no human driver involved. For the design of Level 5 self-driving cars, the auto makers are even removing the gas pedal, brake pedal, and steering wheel, since those are contraptions used by human drivers. The Level 5 self-driving car is not being driven by a human and nor is there an expectation that a human driver will be present in the self-driving car. It’s all on the shoulders of the AI to drive the car.
For self-driving cars less than a Level 5 or Level 4, there must be a human driver present in the car. The human driver is currently considered the responsible party for the acts of the car. The AI and the human driver are co-sharing the driving task. In spite of this co-sharing, the human is supposed to remain fully immersed into the driving task and be ready at all times to perform the driving task. I’ve repeatedly warned about the dangers of this co-sharing arrangement and predicted it will produce many untoward results.
For my overall framework about AI self-driving cars, see my article: https://aitrends.com/selfdrivingcars/framework-ai-self-driving-driverless-cars-big-picture/
For the levels of self-driving cars, see my article: https://aitrends.com/selfdrivingcars/richter-scale-levels-self-driving-cars/
For why AI Level 5 self-driving cars are like a moonshot, see my article: https://aitrends.com/selfdrivingcars/self-driving-car-mother-ai-projects-moonshot/
For the dangers of co-sharing the driving task, see my article: https://aitrends.com/selfdrivingcars/human-back-up-drivers-for-ai-self-driving-cars/
Let’s focus herein on the true Level 5 self-driving car. Much of the comments apply to the less than Level 5 self-driving cars too, but the fully autonomous AI self-driving car will receive the most attention in this discussion.
Here’s the usual steps involved in the AI driving task:
- Sensor data collection and interpretation
- Sensor fusion
- Virtual world model updating
- AI action planning
- Car controls command issuance
Another key aspect of AI self-driving cars is that they will be driving on our roadways in the midst of human driven cars too. There are some pundits of AI self-driving cars that continually refer to a Utopian world in which there are only AI self-driving cars on the public roads. Currently there are about 250+ million conventional cars in the United States alone, and those cars are not going to magically disappear or become true Level 5 AI self-driving cars overnight.
Indeed, the use of human driven cars will last for many years, likely many decades, and the advent of AI self-driving cars will occur while there are still human driven cars on the roads. This is a crucial point since this means that the AI of self-driving cars needs to be able to contend with not just other AI self-driving cars, but also contend with human driven cars. It is easy to envision a simplistic and rather unrealistic world in which all AI self-driving cars are politely interacting with each other and being civil about roadway interactions. That’s not what is going to be happening for the foreseeable future. AI self-driving cars and human driven cars will need to be able to cope with each other.
For my article about the grand convergence that has led us to this moment in time, see: https://aitrends.com/selfdrivingcars/grand-convergence-explains-rise-self-driving-cars/
See my article about the ethical dilemmas facing AI self-driving cars: https://aitrends.com/selfdrivingcars/ethically-ambiguous-self-driving-cars/
For potential regulations about AI self-driving cars, see my article: https://aitrends.com/selfdrivingcars/assessing-federal-regulations-self-driving-cars-house-bill-passed/
For my predictions about AI self-driving cars for the 2020s, 2030s, and 2040s, see my article: https://aitrends.com/selfdrivingcars/gen-z-and-the-fate-of-ai-self-driving-cars/
Returning to the topic of AI self-driving cars and ridesharing, along with the topic of scheduling, let’s consider what the future is likely to present.
I’ve stated in my writings and speeches that the advent of AI self-driving cars will be more so than solely being done as fleets.
As background for you, some pundits claim that no one will individually own an AI self-driving car because such vehicles will be overly expensive. Therefore, in this theory, AI self-driving cars will be owned by the likes of either automakers, tech firms, or ridesharing firms, and be considered as working in collectives that we might call a fleet of AI self-driving cars.
That seems like a rather narrow view of the future.
AI Self-Driving Cars to Become a Flood of Ridesharing
If an automaker or tech firm or ridesharing firm can make a buck off of AI self-driving cars, why wouldn’t individuals seek to do the same?
Those with this other theory are narrow thinking in that they view car ownership as simply and exclusively a cost. Today, when you buy a car, you use it to go to work, and going on vacation, and driving to the store, etc. You aren’t making money by owning the car. It is your means of conveniently getting around.
The advantage of an AI self-driving car is that it comes with a built-in driver (in the case of true Level 5 AI self-driving cars). This means that the AI self-driving car can be used whenever you want, and you don’t need to be the driver, and nor do you need to find or hire a driver. Keeping in mind too that most people only use their car for about 5-10% of the day, a car is a tremendously underutilized asset that can be deployed to your personal and financial well-being.
How could you afford an AI self-driving car if it might cost into the hundreds of thousands of dollars? Easy, by turning it into a money maker. While you are at work, you send your AI self-driving car out to do ridesharing. When you are asleep, you do likewise.
This will create a huge cottage industry of small businesses, whereby you purposely buy an AI self-driving car, likely taking out a loan to cover it, and are anticipating that the revenue generated by the AI self-driving car will make your purchase worthwhile. There is a chance of a solid ROI (Return On Investment) for this approach of buying an expensive asset and putting it to work.
I’m predicting, perhaps boldly, we’ll see a flourishing cottage industry surrounding the advent of AI self-driving cars.
For the affordability of AI self-driving cars, see my article: https://www.aitrends.com/selfdrivingcars/affordability-of-ai-self-driving-cars/
For the non-stop aspects of AI self-driving cars, see: https://www.aitrends.com/selfdrivingcars/non-stop-ai-self-driving-cars-truths-and-consequences/
For my article about the crossing of the Rubicon and AI self-driving cars, see: https://www.aitrends.com/selfdrivingcars/crossing-the-rubicon-and-ai-self-driving-cars/
For recalls that will undoubtedly happen to AI self-driving cars, see my article: https://www.aitrends.com/selfdrivingcars/auto-recalls/
For my article about the future of jobs including AI self-driving car repair services, see: https://www.aitrends.com/selfdrivingcars/future-jobs-and-ai-self-driving-cars/
That being said, I’ve also forewarned that this blossoming might get somewhat out-of-hand.
Besides individuals jumping into the fray, and besides the usual suspects like ridesharing firms and automakers, you might as well add other kinds of firms too. You could be a firm in a completely unrelated industry and see the writing on the wall that money can be made off the backs of AI self-driving cars.
Today’s firms that make money from the utilization of cars for ridesharing have to jump through lots of hoops to do so. Ridesharing firms need to find drivers and keep those drivers happy. No need to do so for an AI system that’s your always available driver, it’s happy already (well, kind of). You can also readily outsource things like the maintenance needed for the self-driving cars and other kinds of logistics aspects.
If my predictions come true, we’ll see a flood of AI self-driving cars that are flowing in and around our streets. This will be the next gold rush.
Let’s consider then the notion of AI self-driving cars roaming around our streets.
Pundits tend to imagine a Utopian world in which you come out to the street and within seconds there is an AI self-driving car there at your beck and call. Sounds great! We will all be able to reduce delay time in getting a ride. Rides on demand.
Yes, that might be true, but how did that AI self-driving car get to you, doing so quickly?
You might have requested it in-advance, perhaps via a mobile app, and then when it arrived, you went outside to get into it for your ride. That’s one way to arrange the ride.
Another involves simply going out to the curb and hailing a ride. I’d dare say most of us are using that method these days. You used to hail a cab by waving frantically at cabs that wandered past you. Now, you use your mobile app to see how far away a ride might be, and once you select it, the driver heads in your direction.
If you choose to use a particular ridesharing service, it means that you are only going to be seeing those available ridesharing cars that are perchance signed-up with that service. There might be other ridesharing services that have available cars and those are even closer to your position at the curb, at that moment, but you tend to ignore them and go with the ridesharing service that you prefer.
Suppose in the future that there are zillions of ridesharing cars that are nearby when you happen to go out to the curb. Rather than being focused on one particular ridesharing firm, you might be willing to go with whichever ridesharing car happens to get there soonest. Of course, you also care about the cost, and the quality of the ride, and let’s assume for the moment that’s a given.
Put on your hat of the firms and individuals that will own AI self-driving cars and are trying to make money by using those self-driving cars as a ridesharing service.
They want to put their AI self-driving car in places that will maximize their revenues of doing ridesharing. This means they want their AI self-driving car to be chosen for a paying fare. They also want to minimize the unused time of their AI self-driving car, which essentially is nonpaying, such as when their AI self-driving car is roaming to find a fare.
If you knew that in say downtown Los Angeles that Wilshire Boulevard will have the greatest number of potential riders between 5 p.m. and 6 p.m. on weekdays, where would you want your AI self-driving car to be?
Well, it’s similar to the drivers at the Tuesday events, namely they wanted to be near the action. You would not want your AI self-driving car roaming five blocks away, since another AI self-driving car that’s roaming on Wilshire Boulevard is likely going to snag the fare that your AI self-driving car might have been able to get, but your AI self-driving car was not in the right place.
The Future of Ridesharing via AI Self-Driving Cars
Here’s what might happen.
All these businesses that have AI self-driving cars are going to want to funnel them into whatever places and at whatever times will earn them the most in fare revenues. Since they all want to do this, you’ll have a grand convergence of AI self-driving cars, all flocking to the same places, ones that seem to offer the most chance of getting riders.
For the Tuesday events, recall that the ridesharing cars and taxis figured out on their own via word-of-mouth that it made sense to hangout at the HQ on Tuesday evenings, along with jockeying for physical position. That’s exactly what’s going to happen with AI self-driving cars that have been put into ridesharing service, which I’d postulate will be most of the AI self-driving cars that will exist on our roadways.
In the downtown Los Angeles quarter, you could have a flood of AI self-driving cars, all jockeying for position. All vying to get those riders. They would swarm like moths to a flame. Those that own those AI self-driving cars don’t really care about the traffic snarl, other than if it reduces their chances of their AI self-driving car not making a buck because of the density of competition.
For the human riders that want a ride, it could be a nirvana of choices. Those AI self-driving cars all coming to get you, and presumably the owners might have setup various special discounts and incentives. Use the XYZ ridesharing AI self-driving car that’s coming down the street right now, and you’ll get a 10% off for picking it, rather than using the ABC ridesharing AI self-driving car that’s this moment pulled to the curb where you are standing. Isn’t it worth the 10% off to wait another 15 seconds for your ride?
I’d like to also take a momentary step back and ask you to contemplate what this kind of flood of AI self-driving cars will do to the traffic situation.
If you have a belly full of AI self-driving cars trying to all circle around and around among a few blocks area in downtown, the resultant impact to traffic movement will be startling. Gridlock will ensue. Those AI self-driving cars don’t care per se about sitting in traffic, which humans tend to avoid. The only thing to curtail the AI self-driving car from sitting in traffic is the opportunity lost of potentially snagging a fare, because the AI self-driving car was stuck in traffic a block from where a rider was seeking to get a ride.
This also brings up another pet peeve of mine.
Pundits keep saying that we won’t need parking lots anymore, due to the emergence of AI self-driving cars. The logic seems to be that your AI self-driving car will roam while you aren’t using it. It won’t park. Instead, it just keeps roaming.
Keep in mind there is a cost involved in having your AI self-driving car roaming. For each minute or hour that it is underway, it is like any car that will be encountering wear and tear. There is also the cost of the electrical power that it is consuming, if an EV, or the cost of gasoline if conventionally powered. This constant roaming is not cost free.
And, as per my earlier comments, your roaming AI self-driving car is going to be plugging up traffic. All of those roaming AI self-driving cars are going to mean more cars on tight city streets. I dare say that most pundits don’t seem to realize what this continual roaming will do. I’ll also mention that this continual roaming will have a damaging effect on the roadways, which is another cost that needs to be considered (at least for the entities that maintain the roadway infrastructure).
My view is that we are going to need to have waiting areas for AI self-driving cars, essentially parking lots. This would be akin to what you see done at airports. There are ridesharing and taxi waiting areas that are parking lots, though the cars might be moving slowly or waiting in line, and they gradually are released from the waiting area to go to where they can pick-up fares.
Some pundits would say that sure, go ahead and create those waiting areas (parking lots), but put them further away from say the downtown area. You can maybe get cheaper land outside of the downtown, and it can be unused property that nobody otherwise wants to use (abandoned cow pastures made into a waiting area?), since it isn’t in downtown, and have the roaming AI self-driving cars sit there.
I ask you, how does that square with the idea that the owners of those AI self-driving cars want their self-driving cars to be in the places that will maximize their fares?
If my AI self-driving car is sitting in a waiting area that’s twenty minutes from downtown Los Angeles, it is not going to make much money, especially in comparison to a competitor that has their AI self-driving car driving around and around in the downtown streets to snag fares. Plus, even if it gets a reservation to go pick-up a fare, you now have the cost of the AI self-driving car going the twenty minutes from the waiting area to downtown (and, the cost of when the AI self-driving car went to the waiting area to begin with).
My point being that it seems doubtful to believe that you can relinquish back all of the parking lots in congested areas by either betting on roaming AI self-driving cars or by thinking that you’ll simply relegate the AI self-driving cars to sitting outside of town in some non-congested place and waiting to be hailed.
We all need to be thinking more clearly about these matters. Shortcuts as a way of thinking is going to make for larger problems.
Precision Scheduling of Autonomous and Human-Based Ridesharing (PSAHBR)
I had mentioned that the Tuesday evening desperation of taxis and ridesharing services was somewhat dealt with by doing things in a more planned way. Similarly, the freight train hubs are going to be transformed into a more rigorous and systematic form of coordination, via the PSR (Precision Scheduling Railroading).
One solution to the AI self-driving car flood of ridesharing might be to consider putting in place a kind of universal Precision Scheduling of Autonomous and Human-Based Ridesharing (PSAHBR) system.
In essence, ridesharing services would put into inventory of this universal scheduling system their AI self-driving cars as an available ridesharing vehicle. The system would then try to schedule the placement of the AI self-driving cars to meet demand.
It will be a complicated algorithm, that’s for sure.
In a manner of speaking, it is reminiscent of the National Resident Matching Program (NRMP), often referred to as The Match, which occurs in the United States and involves the matching of U.S. medical school students into the available residency programs at teaching hospitals each year. A non-profit non-governmental entity was set up to do this. If you aren’t aware of it, you might want to look online about the matter, and it uses a famous problem known as the “stable marriage problem” as an underlying way to find an algorithm to deal with the matching process.
The mighty PSAHBR would be a kind of matching that involves the pairing of those seeking a ride with an available ridesharing car. Notice that I did not say that it would necessarily be an AI self-driving ridesharing car that is only in the inventory of the PSAHBR system.
As I’ve mentioned earlier, we are going to have a mixture of human driven cars and AI self-driving cars for quite a while. If you were to design the PSAHBR for solely dealing with the assignment of AI self-driving cars, it would mean that the human driven ridesharing cars would not be included.
Those human driven ridesharing cars could then potentially poach the rides that the AI self-driving cars are trying to get. Or, you would have a backlash of the human driven ridesharing cars that those human drivers are being discriminated against by the AI self-driving car availabilities, and perhaps the human drivers weren’t able to get rides that were instead being handed to the AI self-driving cars.
Presumably, the PSAHBR would smoothen out the traffic situation and aim to reduce the continual and somewhat wasteful aspects of ridesharing roaming, whether by human drivers or by AI self-driving cars. The system would need to have an indication of where riders tend to want rides, and by using Machine Learning and Deep Learning could try to predict when rides are needed, along with figuring out optimal ways to arrange for the ridesharing inventory to be available at the right places at the right times.
One question right away that one needs to ask involves whether those that own the ridesharing cars are going to voluntarily seek to use such a system. It all depends.
If the PSAHBR can do a good enough job of scheduling, it would imply that the owners of the ridesharing services will earn more revenue and have less cost than if they had tried to just let their ridesharing cars roam. Obviously, the owners would be making a decision about whether it is better to go free or to use the system.
That being the case, there might be localities that decide to force the ridesharing services to use such a system. Akin to my earlier indication about airports, the airport authority is able to ban ridesharing from freely entering into the airport and force the ridesharing services to comply to the rules that are established. Presumably, a city could do likewise.
Who would put in place the PSAHBR?
It could be a non-profit non-governmental entity that was established to create and keep in shape such a system.
Or, it could be a governmental agency that opts to do so.
One would certainly expect that the major ridesharing services would be tending to craft something like this anyway, if nothing else to try and watch over their own fleets. Would other fleets join in?
Would the mom-and-pop cottage industry join in?
Likely it would depend upon the perceived “fairness” of how the ridesharing cars are given fares.
For more about regulations in AI self-driving cars, see my article: https://www.aitrends.com/selfdrivingcars/assessing-federal-regulations-self-driving-cars-house-bill-passed/
For my article about how induced demand will impact the future of AI self-driving cars, see: https://www.aitrends.com/selfdrivingcars/induced-demand-driven-by-ai-self-driving-cars/
For the potential of an invasion of AI self-driving cars, see my article: https://www.aitrends.com/selfdrivingcars/invasive-curve-and-ai-self-driving-cars/
For my article about predicting the future of AI self-driving cars, see: https://www.aitrends.com/ai-insider/key-equation-for-predicting-year-to-prevalence-for-ai-self-driving-cars/
Conclusion
Whatever does happen in the future, I think it is a reasonable bet that once AI self-driving cars become prevalent, there will be a swirling of ridesharing that will make our heads spin. At first, it might seem like a welcomed capability. After things turn ugly due to the over abundance of ridesharing, there will be a wringing of the hands about what to do.
The public and the regulators are likely to realize that something needs to be done, once traffic snarls emerge and there is a cutthroat vying for fares.
Can someone get all of the ridesharing services to voluntarily come together into a universal scheduling system, or will it require a more heavy hand to do so?
Time will tell.
Meanwhile, for those of you that are interested in developing new and innovative apps, consider the kind of scheduling system that the PSAHBR would be, and get coding.
Copyright 2020 Dr. Lance Eliot
This content is originally posted on AI Trends.
[Ed. Note: For reader’s interested in Dr. Eliot’s ongoing business analyses about the advent of self-driving cars, see his online Forbes column: https://forbes.com/sites/lanceeliot/]
AI in Space: NASA Studying Exoplanets, ESA Supporting Satellites
By AI Trends Staff
AI is being employed in a wide range of efforts to explore and study space, including the study of exoplanets by NASA, the support of satellites by ESA, development of an empathic assistant for astronauts and efforts to track space debris.
NASA scientists are partnering with AI experts from companies including Intel, IBM and Google to apply advanced computer algorithms to problems in space science.
Machine learning is seen as helping space scientists to learn from data generated by telescopes and observatories such as the James Webb Space Telescope, according to a recent account from NASA. “These technologies are very important, especially for big data sets and in the exoplanet field,” stated Giada Arney, an astrobiologist at NASA’s Goddard Space Flight Center in Greenbelt, Md. (Exoplanets are beyond the solar system.) “Because the data we’re going to get from future observations is going to be sparse and noisy, really hard to understand So using these kinds of tools has so much potential to help us.”
NASA has laid some groundwork for collaborating with private industry. For the past four summers, NASA’s Frontier Development Lab (FDL) has brought together technology and space innovators for eight weeks every summer to brainstorm and develop code. The program is a partnership between the SETI Institute and NASA’s Ames Research Center, both located in Silicon Valley.
The program pairs science and computer engineering early-career doctoral students with experts from the space agency, academia and some big tech companies. The companies contribute hardware, algorithms, supercomputing resources funding, facilities and subject matter experts. Some of the resulting technology has been put to use, helping to identify asteroids, find planets and predict extreme solar radiation events.
Scientists at Goddard have been using different techniques to reveal the chemistry of exoplanets, based on the wavelengths of light emitted or absorbed by molecules in their atmospheres. With thousands of exoplanets discovered so far, the ability to make quick decisions about which ones deserve further study would be a plus.
Arney, working with Shawn Domagal-Goldman, an astrobiologist at Goddard Center, working with technical support from Google Cloud, deployed a neural network to compare performance to a machine learning approach. University of Oxford computer science graduate student Adam Cobb led a study to test the capability of a neural network against a widely-used machine learning technique known as a “random forest.” The team analyzed the atmosphere of WASP-12b, an exoplanet discovered in 2008 that had a comparison study done with a random forest technique, using data supplied by NASA’s Hubble Space Telescope.
“We found out right away that the neural network had better accuracy than random forest in identifying the abundance of various molecules in WASP-12b’s atmosphere,” Cobb stated. Beyond the greater accuracy, the neural network model could also tell the scientists how certain it was about its prediction. “In a place where the data weren’t good enough to give a really accurate result, this model was better at knowing that it wasn’t sure of the answer, which is really important if we are to trust these predictions,” states Domagal-Goldman.
The European Space Agency (ESA) is studying how to employ AI to support satellite operations, including relative position, communication and end-of-life management for large satellite constellations, according to an account from ESA.
The ESA has engaged in a number of studies on how to use AI for space applications and spacecraft operations as part of its Basic Activities program. One study examines using AI to support autonomous spacecraft that can navigate, perform telemetry analysis and upgrade their own software without communicating with Earth.
Another study focused on how AI can support the management of complex satellite constellations, to reduce the active workload of ground operators. Greater automation, such as for collision avoidance, can reduce the need for human intervention.
Additional studies are researching how a swarm of picosatellites – very small ones – can evolve a collective consciousness. The method employed explored known results in crystallography, the study of crystals, which may open a new way of conceiving lattice formation, a sub-discipline of order theory and abstract algebra.
AI Helping Astronauts Too; An AI Assistant with Empathy Coming
Astronauts traveling long distances for extended periods might be offered assistance from AI-powered emotional support robots, suggests a recent report in yahoo! News. Scientists are working to create an AI assistant that can sense human emotion and “respond with empathy.”
The robots could be trained to anticipate the needs of crew members and “intervene if their mental health is at stake.” An AI assistant with empathy could be helpful to astronauts on a deep-space mission to Mars.
Astronauts on the International Space Station have an intelligent robot called CIMON that can interact, but is lacking in emotional intelligence. NASA CTO Tom Soderstrom has stated. A team at the organization’s Jet Propulsion Laboratory is working on a more sophisticated emotional support companion that can help fly the spacecraft as well as track the health and well-being of crew members.
AI Employed in Effort to Track Space Debris
Space debris is becoming a critical issue in space. Scientists count more than 23,000 human-made fragments larger than 4 inches, and another 500,000 particles between half an inch and 4 inches in diameter. These objects move at 22,300 miles per hour; collisions cause dents, pits or worse.
Scientists have begun to augment the lasers employed to measure and track space debris with AI, specifically neural nets, according to a recent account in Analytics India Magazine.
Laser ranging technology was becoming a challenge due to poor prediction accuracy, small size of objects, and no reflection prism on the surface of debris, making it difficult to spot the exact location of fragments. Scientists began using a method to correct the telescope pointing error of the laser ranging system by enhancing certain hardware equipment. Most recently, AI deep learning techniques are starting to be employed to enhance the correction models.
Chinese researchers from the Chinese Academy of Surveying and Mapping, Beijing and Liaoning Technical University, Fuxin have worked to enhance the accuracy of identifying space junk. The team used a backpropagation neural network model, optimized by a proposed genetic algorithm and the Levenberg-Marquardt algorithm (used in curve fitting) to help pinpoint the location of debris. The results showed higher probability of accurately locating debris between three and nine times.
“After improving the pointing accuracy of the telescope through deep learning techniques, space debris with a cross-sectional area of one meter squared and a distance of 1,500 kilometres can be identified,” stated Tianming Ma of the Chinese Academy of Surveying and Mapping, Beijing and Liaoning Technical University, Fuxin.
Read the source articles from NASA, the ESA, Robotics Business Review, in yahoo! News and in Analytics India Magazine.
Barclays Innovating in use of AI in Banking
By AI Trends Staff
Barclays Bank is emerging as an innovator in the use of AI in financial services. The UK bank, ranked 20th on the S&P Global’s list of the top 100 banks, works with suppliers of AI products and services more than it develops AI applications in house, according to a recent account from emerj.
Here are three AI initiatives underway at Barclays and the industry partners working on each one:
- Risk Modeling with Simudyne, employs predictive analytics to assess loan risk
- Voice Recognition for Authentication, with Nuance, aims to apply verification and authentication using voice recognition;
- Business Process Automation with IBM, a project to automate debit card deactivation, and analyze customer feedback.
London-based Simudyne, founded in 2007, offers simulation software that models complex transactions with large data sets. The company uses “agent-based modeling” to execute banking tasks using simulated transactions and customers, resulting in detailed predictions that aim to improve on past practice. Banking customers try to better understand the upside and downside risks associated with making a certain loan or investment.
Barclays plans to use Simudyne to analyze mortgages, conduct stress tests, study investment opportunities and analyze low liquidity scenarios.
Jes Staley, CEO of Barclays, is quoted on the Simudyne website stating, “Simudyne is ground-breaking technology currently being leveraged across Barclays and enables us to model multiple scenarios on huge datasets, so we can understand our risk, exposure and options.”
Barclays plans to run the Simudyne simulations in the cloud, which positions the bank to tap wider sets of data from any of the bank’s data science labs in the hopes of getting an early warning on shifts in banking trends.
Nuance, providing speech recognition and AI, has helped Barclays build a system to authenticate customers using voice over the phone. Nuance offers “voice biometrics” based on natural language processing technology. After the system was implemented, the bank saw a 90 percent reduction in complaints about the security questions the bank had to ask to authenticate customers. The bank also saw a 15 percent reduction in average call times.
Anne Grim, head of Global Client Experience for Barclays Wealth and Investment Management unit, states in a case study on Barclays on the Nuance website, “The use of Nuance’s voice biometric technology has been integral in our mission to deliver an excellent customer experience.”
IBM is providing its Business Process Manager and Blueworks Live products to help Barclays raise customer satisfaction through techniques such as offering fraud alerts via SMS messages, monitor possible identity theft and streamline processing of lost or stolen credit cards.
The system is said to have allowed the bank to replace missing or compromised cards 67% ages, and roll out new business processes 88% faster.
Barclays Eagle Labs unit works with startups and small businesses to help grow the UK economy. An AI at Barclays section on the Eagle Labs website to promote collaboration on AI projects throughout the organization and with its customers. The bank started its “AI Frenzy” events in 2018, inviting industry specialists and the bank’s own experts. The events have also been hosted by universities as a way to help students get introduced to AI and machine learning.
Investment in Simudyne
Barclays entered a closer relationship with Simudyne in 2019, investing $6 million to fund expansion, according to an account in Citya.m. “Partnering with high-growth fintech companies like Simudyne is core to our technology strategy,” stated Andy Challis, managing director of principal investments at Barclays, on the announcement of the investment. “As its adoption becomes more widespread, Simudyne’s platform will ultimately help cultivate a stronger, more efficient tech-enabled financial services sector.”
Writing about the relationship on the Barclays site, C.S. Venkatakrishnan, Barclays Group Chief Risk Officer, emphasized that the agent-based modeling approach has matured and been enabled by a more powerful generation of hardware. This is allowing the bank to create more realistic simulations to account for feedback loops, relationships between agents and complex scenarios that include external factors such as climate impact.
“Simudyne’s platform simplifies the design and operation of models by leveraging the power of distributed computing in the Cloud,” Venkatakrishnan stated. “This allows them to be deployed and scaled on-demand, radically reducing operational costs.”
Justin Lyon, CEO and Founder of Simudyne, stated in the Barclays case study, “Agent-based modelling has opened up a number of opportunities for banks to test any decision before committing resources or taking any action in the physical world. Barclays’ decision to use simulation as a competitive advantage is just one expression of their focus on innovation.”
Read the source article and reports at emerj, Eagle Labs, Citya.m and at Simudyne.
AI Being Incorporated in Architecture, Is Potentially Transforming
By John P. Desmond, AI Trends Editor
AI is being incorporated into architecture in a range of ways today, from evaluations of the efficiency of construction to simulations of human movement in the space. Some examples of how AI is being incorporated into architecture shed light on where things are now and where they are headed.
Philips Lighting Headquarters in Eindhoven, Netherlands, is experimenting with lighting as the center point of a design project, according to a recent account in archdaily.com. The project proposal describes a parametric designed “tree” which is made up of 1,500 “leaves” that hang suspended from the ceiling. AI is incorporated in the programming of the panels to deliver different lighting scenarios in a non-repetitive way.
The Amaro Stores in Brazil were founded on a concept of combining in-person physical shopping and an online experience. The Sao Paulo-based firm, founded in 2012 as an e-commerce website, started opening “Guide Shops” in Rio De Janeiro and São Paulo, giving shoppers a closer connection to their designers, by being able to browse new collections online, according to an account in Brazil Reports. Clothes can be ordered in the store and will be delivered in two days.
Amaro uses data collection techniques on social media via a network of thousands of “influencers,” an effort to provide customers what they want. The new Guide Shop in Sao Paulo employs AI technology in the store in the use of cameras that capture a customer’s profile, approximate age, reactions and responses to simulations through facial recognition.
Greenest Building in Shanghai
An architectural design competition for an office building incorporating intelligence to help deliver on sustainability goals was won by Zaha Hadid Architects for a design of three office towers surrounding a park, according to an account on the website of Archello, supplier of an architecture and design platform. The complex of smart buildings will include shopping, dining and leisure facilities. It is located adjacent to the Yangpu Bridge on the Huangpu River, and will set new benchmarks for energy conservation, energy efficiency and sustainability.
Each building will have: integrated thermal mass to reduce heating and cooling requirements, high-efficiency heating and ventilation with waste heat recovery, cooling systems using non-potable water, and thermal ice storage for cooling. The ice used will be created on site by chillers at night that use off-peak electricity stored in thermal tanks. Solar panels on the roof and incorporated into the south facing facades will be connected to battery storage via a micro-grid. The projection is for reduced energy consumption of 25%.
A management system will monitor the interior environment and automatically react to variations in temperature, air quality, natural daylight, or number of occupants. The smart system collects data to predict and optimize energy usage. Locally produced prefabricated components will further reduce the building’s carbon footprint. Procurement will prioritize the use of recycled materials.
Greenhouse on Top Floors of Intesa Sanpaolo
The design firm Renzo Piano designed and built the high-rise headquarters of Italian mega bank Intesa Sanpaolo in Turin, in what the architect called a “bioclimatic” building. The 38-story tower beside a public park is 545 feet tall, and has many eco-friendly features, according to an account in Architect. The top floors are occupied by a triple-height greenhouse, with views over Turin and the snow-capped mountains outside the city. Photovoltaic cells cover the tower’s southern facade, and the stairwell on that side of the building is also a “vertical winter garden” with hanging plants beside the steps.
The tower, which cost some $565 million to build and took more than seven years to complete, has a number of passive heating and cooling elements. In the summer, apertures in the exterior wall draw cool air into cavities between the concrete-slab floor, which radiantly cool the offices during the day. The ceilings are 10 feet 6 inches high, to allow natural light to penetrate deep into the building’s interior. Computer-controlled shades keep out the sun on the brightest days. And the tower has a geothermal heating and cooling system.
New approaches to incorporating AI into architecture include incorporation within design tools. “Parametric architecture” refers to a design system that allows the architect to adjust system parameters to produce different outputs and create forms and structures that otherwise would not have been possible, according to a recent account in Interesting Engineering. The tools give the architect the ability to pick the design output, set the constraints, plug in data and create many iterations of the product or building in minutes. Used within CAD tools such as Grasshopper, the parametric architecture uses geometric programming, with complex algorithms to allow the architect to optimize the building.
Radical Transformation of Architecture Seen
A recent paper written by Harvard Graduate School of Design student Stanislas Chaillou predicted AI is on the verge of radically transforming architecture. Now an architect and data scientist at Spacemaker in Cambridge, Mass., Chaillou had published his paper in Towards Data Science.
“AI will soon massively empower architects in their day-to-day practice,” he stated. He encouraged architects to engage with AI and data scientists to consider architecture as a field of investigation. As guidance, he suggests a “grey box” approach as opposed to a “black box” approach, in which the design pipeline is broken into discrete steps with the user intervening along the way, to maintain tight control. “His tight control over the machine is his ultimate guarantee of the design process quality,” Chaillou stated.
Read the source articles and references at archdaily.com, Brazil Reports, at Archello, in Architect, Interesting Engineering and in Towards Data Science.
AI in COVID-19 Fight: Pope Issues Ethical Challenge; Voice Studied to Help in Detection
By AI Trends Staff
The worldwide fight against COVID-19 continues to challenge AI experts. The Pope issued a challenge for AI experts to develop an “ethical algorithm” that would ensure fairness; Some new AI research shows how people are feeling about the virus. Other researchers are experimenting with the use of sound to detect the virus.
Shortly before the Vatican closed due to the virus, members of the Pontifical Academy for Life, which researches bioethics and Catholic moral theology, worked on getting a commitment from AI developers to write an “ethical” algorithm in each AI system, according to an account in SSPX.news, the communication agency of the Society of St. Pius, based in Paris.
“Following the example of electricity, AI is not necessary to perform a specific action, it is rather intended to change the way, the mode with which we carry out our daily actions,” stated Fr. Paolo Benanti, a professor of moral theology and bioethics at the Pontifical Gregorian University in Rome. He spoke at the conference held Feb. 26 and 27, 2020, on what he sees at stake in the digital revolution represented by AI.
As AI learns, ingests more data, it becomes more powerful and poses a bigger moral problem.
The moral problem only becomes bigger: “When the machine replaces man in decision-making, what kind of certainty would we have to let the machine choose who should be treated or not, and how? On what basis should we allow a machine to designate which of us is trustworthy and who is not?” Fr. Benanti stated.
The Academy is recommending development of the “good” algorithm. “If we want the machine to support man and the common good, without ever replacing the human being, then the algorithms must include ethical values, not just data,” says Fr. Benanti.
As the fight against COVID-19 continues, the future will tell whether “ethical algorithms” can be written and applied within related AI systems.
Natural Language Understanding Applied to Social Media
A company specializing in natural language reading and semantics is using AI to extract emotions and sentiment from 63,000 social media posts in English on Twitter for every 24-hour period, creating a semantic analysis of people’s feelings during the spread of the COVID-19 virus worldwide, according to an account in Forbes. The company, Expert System of Modena, Italy, has over 20 years of experience in natural language understanding. The system analyzes data every 24 hours and has it interpreted by Sociometrica, a research and publishing firm based in Spain.
The data shows anxiety is high and fear is widespread, with anxiety around the virus crisis dominating communications, stated Walt Mayo, CEO of Expert System. “We also have seen growing criticism of individual behavior that is considered irresponsible and goes against advice to follow social distancing and other recommendations to “flatten the curve,” he added. “But we also have seen growing expressions of gratitude toward health care workers and emerging signs of hope more broadly.”
Success of social distancing strategies to slow the spread of the virus depend on the behavior of individuals, so Mayo vouches for the importance of monitoring sentiment changes.
Sound Research to Detect Virus Underway
Another group of AI researchers is studying whether the sound of a person’s voice can be used to detect the presence of a COVID-19 infection. The researchers are collecting voice samples from people who have contracted the virus, to train the system to recognize patterns in breathing, coughing and speech, according to an account in The Wall Street Journal.
Researcher Tomas Teijeiroand his colleagues at the École Polytechnique Fédérale de Lausanne (the Swiss Federal Institute of Technology) are working on a project called Coughvid. They put out an appeal for people diagnosed as infected with COVID-19 provide coughing samples via their website. Some 800 samples have been collected so far. When the app is available, people install it, cough and the result appears immediately.
A similar effort is underway between Voca.ai, an Israeli startup that has developed virtual call-center agent software, and Carnegie Mellon University to collect voice samples of people reciting the alphabet and making other sounds. “We believe that a lot of information could be perceived through speech, and this disease as well, we believe, can be identified by speech,” stated Alan Bekker, co-founder and chief technology officer of Voca.ai. Early results showed 75 percent accuracy in detecting the virus; the hope is accuracy is improved as more data is added.
Researchers at New York University who work with Facebook Inc. chief AI scientist Yann LeCun are gathering samples of people taking breaths into their phones—an effort they dub “breathe for science.” And the Seattle Fire Department is working with a Danish startup called Corti to test an AI tool that analyzes breathing and coughing in 911 calls.
Backers of the audio projects say they plan to share their data with other researchers. The Voca.ai project had collected about 1,450 samples as of mid-April; the breaking project has about 500. Bekker of Vocas.ai stated 5,000 samples from people known to have the virus would allow for a more full analysis of the data.
In other AI and COVID-19 related developments:
DeviceBits has announced several AI- and Chatbot-based support resources for companies and their call center partners to ensure proper service levels remain for end user customers during the global COVID-19 outbreak and containment efforts. In order to maintain proper customer service levels, and to support the increasing remote workforce population for call center staff, DeviceBits has made available COVID-19-specific AI and ChatBot-enabled solutions that offer self-support for general and high-volume customer questions; intelligent agent assistance with live agent chat and messaging; and an AI-powered chatbot with transfer functionality for special technical issues. DeviceBits COVID-19 quick start resources immediately available include Academy, a self-support web-based knowledge center to provide COVID-19 quick answers and deflect call volumes during this challenging time; CareAssist+ chime offering intelligent agent assistance, live agent chat and messaging to scale agents more efficiently and train agents in a fraction of the time; and BOT+ chime, AI-powered chatbot with transfer functionality to automate customer requests and provide coverage for off-hour support. More information.
Lazarus AI, makers of a product combining AI and deep learning technology for early cancer detection, is spearheading an initiative to gather data needed by medical professionals to assess whether someone with some symptoms should be tested for the COVID-19 virus. Called CATCH, for Covid Assessment Tool for Community Health, the tool has been built by a volunteer group of AI specialists spanning many disciplines. The link is to a free survey people with systems take online; the results can be sent to a physician the patient chooses.
The tool is secure and HIPAA compliant, built for a single purpose according to CDC recommendations and with extensive input from emergency doctors. The survey will help patients understand the severity of their systems. “Local hospital systems will now have a clearer picture of emerging patient surges and the tools necessary to prioritize more effectively, and we will stand a better chance of weathering this together,” stated Ariel Elizarov, CEO and founder of Lazarus in Cambridge, Mass. Learn more at CATCH.
Lore IO, providers of an AI-powered common data model that enables unified data views and faster vendor onboarding, has announced its free COVID-19 Data Onboarding Initiative, a program designed to help pharmaceutical organizations shorten development cycles to speed drug readiness. The program allows organizations to onboard data from up to three unique sources in only 30 days instead of months which is the time it typically takes using traditional methods. Lore IO’s COVID-19 Data Onboarding Initiative focuses on expediting the onboarding and transformation of three vendor sources of industry data, which allows the user to create a unified view of the data for analysis and accelerate the execution of their go-to-market plan. More information.
Using data from Elsevier’s PharmaPendium product, which includes searchable FDA/EMA drug approval documents, as well as pharmacokinetic and efficacy data, ExactCure, a personalized medicine startup that uses AI technology to reduce medication errors, is developing personalized model simulations that will provide information to physicians to improve the dosing of COVID-19 related therapies. PharmaPendium will provide ExactCure with pharmacokinetic information for approximately 20 approved drugs that have been widely cited in the literature and the news, such as Hydroxychloroquine, Chloroquine, Lopinavir/Ritonavir and Azithromycin, including their regulatory-approval datasets. ExactCure will use this data to build drug-specific exposure models that allow the prediction of pharmacokinetic properties (e.g. Cmax, AUC, Tmax etc). Press release.
A large collaborative effort, led by researchers at U. S. Department of Energy’s Argonne National Laboratory, is combining artificial intelligence with physics-based drug docking and molecular dynamics simulations to rapidly home in on the most promising molecules against COVID-19 to test in the lab. The project is using several of the most powerful supercomputers on the planet—including those at the Texas Advanced Computing Center (TACC), Summit at Oak Ridge National Laboratory, Theta at the Argonne Leadership Computing Facility, and the San Diego Supercomputing Center—to run millions of simulations, train the machine learning system to identify the factors that might make a given molecule a good candidate, and then further explore the most promising results. The team is currently exploring the COVID-19 main protease and will soon begin work on larger proteins that are more challenging to compute but may prove important (e.g., simulation of an all-atom model of the entire virus, which is being produced on the Frontera supercomputer at TACC). The work uses DeepDriveMD (Deep-Learning-Driven Adaptive Molecular Simulations for Protein Folding), a toolkit jointly developed by researchers at Argonne and Rutgers University/Brookhaven National Laboratory. Press release.
Diagnostic healthcare company Nanosticks will be developing and validating ClarityDX COVID-19, a blood test to predict the severity of COVID-19 disease. Patient samples will be collected by its collaborators, the Canadian BioSample Repository at the University of Alberta and Florida-based Century Clinical Research Institute. The plan is to use the ClarityDX biomarker platform to measure viral load along with other known immunological COVID-19 severity risk factors, with the resulting blood test using machine learning algorithms to determine the risk level in SARS-CoV-2-positive patients. Press release.
Thursday, 30 April 2020
Faster SQL Queries on Delta Lake with Dynamic File Pruning
There are two time-honored optimization techniques for making queries run faster in data systems: process data at a faster rate or simply process less data by skipping non-relevant data. This blog post introduces Dynamic File Pruning (DFP), a new data-skipping technique enabled by default in Databricks Runtime 6.1, which can significantly improve queries with selective joins on non-partition columns on tables in Delta Lake.
In our experiments using TPC-DS data and queries with Dynamic File Pruning, we observed up to an 8x speedup in query performance and 36 queries had a 2x or larger speedup.
The Benefits of Dynamic File Pruning
Data engineers frequently choose a partitioning strategy for large Delta Lake tables that allows the queries and jobs accessing those tables to skip considerable amounts of data thus significantly speeding up query execution times. Partition pruning can take place at query compilation time when queries include an explicit literal predicate on the partition key column or it can take place at runtime via Dynamic Partition Pruning.
Delta Lake on Databricks Performance Tuning
In addition to eliminating data at partition granularity, Delta Lake on Databricks dynamically skips unnecessary files when possible. This can be achieved because Delta Lake automatically collects metadata about data files managed by Delta Lake and so, data can be skipped without data file access. Prior to Dynamic File Pruning, file pruning only took place when queries contained a literal value in the predicate but now this works for both literal filters as well as join filters. This means that Dynamic File Pruning now allows star schema queries to take advantage of data skipping at file granularity.
| Per Partition | Per File (Delta Lake on Databricks only) | |
| Static (based on filters) | Partition Pruning | File Pruning |
| Dynamic (based on joins) | Dynamic Partition Pruning | Dynamic File Pruning (NEW!) |
How Does Dynamic File Pruning Work?
Before we dive into the details of how Dynamic File Pruning works, let’s briefly present how file pruning works with literal predicates.
Example 1 – Static File Pruning
For simplicity, let’s consider the following query derived from the TPC-DS schema to explain how file pruning can reduce the size of the SCAN operation.
-- Q1
SELECT sum(ss_quantity)
FROM store_sales
WHERE ss_item_sk IN (40, 41, 42)
Delta Lake stores the minimum and maximum values for each column on a per file basis. Therefore, files in which the filtered values (40, 41, 42) fall outside the min-max range of the ss_item_sk column can be skipped entirely. We can reduce the length of value ranges per file by using data clustering techniques such as Z-Ordering. This is very attractive for Dynamic File Pruning because having tighter ranges per file results in better skipping effectiveness. Therefore, we have Z-ordered the store_sales table by the ss_item_sk column.
In query Q1 the predicate pushdown takes place and thus file pruning happens as a metadata-operation as part of the SCAN operator but is also followed by a FILTER operation to remove any remaining non-matching rows.
When the filter contains literal predicates, the query compiler can embed these literal values in the query plan. However, when predicates are specified as part of a join, as is commonly found in most data warehouse queries (e.g., star schema join), a different approach is needed. In such cases, the join filters on the fact table are unknown at query compilation time.
Example 2 – Star Schema Join without DFP
Below is an example of a query with a typical star schema join.
-- Q2
SELECT sum(ss_quantity)
FROM store_sales
JOIN item ON ss_item_sk = i_item_sk
WHERE i_item_id = 'AAAAAAAAICAAAAAA'
Query Q2 returns the same results as Q1, however, it specifies the predicate on the dimension table (item), not the fact table (store_sales). This means that filtering of rows for store_sales would typically be done as part of the JOIN operation since the values of ss_item_sk are not known until after the SCAN and FILTER operations take place on the item table.
Below is a logical query execution plan for Q2.
As you can see in the query plan for Q2, only 48K rows meet the JOIN criteria yet over 8.6B records had to be read from the store_sales table. This means that the query runtime can be significantly reduced as well as the amount of data scanned if there was a way to push down the JOIN filter into the SCAN of store_sales.
Example 3 – Star Schema Join with Dynamic File Pruning
If we take Q2 and enable Dynamic File Pruning we can see that a dynamic filter is created from the build side of the join and passed into the SCAN operation for store_sales. The below logical plan diagram represents this optimization.
The result of applying Dynamic File Pruning in the SCAN operation for store_sales is that the number of scanned rows has been reduced from 8.6 billion to 66 million rows. Whereas the improvement is significant, we still read more data than needed because DFP operates at the granularity of files instead of rows.
We can observe the impact of Dynamic File Pruning by looking at the DAG from the Spark UI (snippets below) for this query and expanding the SCAN operation for the store_sales table. In particular, using Dynamic File Pruning in this query eliminates more than 99% of the input data which improves the query runtime from 10s to less than 1s.
Without dynamic file pruning
With dynamic file pruning
Enabling Dynamic File Pruning
DFP is automatically enabled in Databricks Runtime 6.1 and higher, and applies if a query meets the following criteria:
- The inner table (probe side) being joined is in Delta Lake format
- The join type is INNER or LEFT-SEMI
- The join strategy is BROADCAST HASH JOIN
- The number of files in the inner table is greater than the value for spark.databricks.optimizer.deltaTableFilesThreshold
DFP can be controlled by the following configuration parameters:
- spark.databricks.optimizer.dynamicPartitionPruning (default is true) is the main flag that enables the optimizer to push down DFP filters.
- spark.databricks.optimizer.deltaTableSizeThreshold (default is 10GB) This parameter represents the minimum size in bytes of the Delta table on the probe side of the join required to trigger dynamic file pruning.
- spark.databricks.optimizer.deltaTableFilesThreshold (default is 1000) This parameter represents the number of files of the Delta table on the probe side of the join required to trigger dynamic file pruning.
Note: In the experiments reported in this article we set spark.databricks.optimizer.deltaTableFilesThreshold to 100 in order to trigger DFP because the store_sales table has less than 1000 files
Experiments and Results with TPC-DS
To understand the impact of Dynamic File Pruning on SQL workloads we compared the performance of TPC-DS queries on unpartitioned schemas from a 1TB dataset. We used Z-Ordering to cluster the joined fact tables on the date and item key columns. DFP delivers good performance in nearly every query. In 36 out of 103 queries we observed a speedup of over 2x with the largest speedup achieved for a single query of roughly 8x. The chart below highlights the impact of DFP by showing the top 10 most improved queries.
Many TPC-DS queries use a typical star schema join between a date dimension table and a fact table (or multiple fact tables) to filter date ranges which makes it a great workload to showcase the impact of DFP. The data presented in the above chart explains why DFP is so effective for this set of queries — they are now able to reduce a significant amount of data read. Each query has a join filter on the fact tables limiting the period of time to a range between 30 and 90 days (fact tables store 5 years of data). DFP is very attractive for this workload as some of the queries may access up to three fact tables.
Getting Started with Dynamic File Pruning
Dynamic File Pruning (DFP), a new feature in Databricks Runtime 6.1, can significantly improve the performance of many queries on Delta Lake. DFP is especially efficient when running join queries on non-partitioned tables. The better performance provided by DFP is often correlated to the clustering of data and so, users may consider using Z-Ordering to maximize the benefit of DFP. To leverage these latest performance optimizations, sign up for a Databricks account today!
--
Try Databricks for free. Get started today.
The post Faster SQL Queries on Delta Lake with Dynamic File Pruning appeared first on Databricks.
Call for User Stories - Jenkins is the Way
One of the things we loved about going to developer conferences was meeting Jenkins users — newbies and old-timers alike — who are excited to talk about their projects and share tips on how to move forward using Jenkins. Since the coronavirus pandemic, we’re learning to rely more on new ways to gather, and it’s happening via Jenkins online meetups, GitHub collaborations, and Twitter threads, to name a few.
It’s a significant change. But what hasn’t changed is the need to share stories about the things users have built, the solutions they’ve developed, and the excellent results they’re getting from some really innovative Jenkins implementations. Then we wondered, why isn’t anyone collecting these user stories and sharing them with the Jenkins community.
Introducing Jenkins is the Way
So we took the first step to record and archive all the great stuff everyone in our community is building with Jenkins. This way, Jenkins users old and new can come to an archive and search for Jenkins solutions for inspiration. We foresee a vast library of solutions from all around the world, solving a wide array of challenges in every industry imaginable. We decided to call this archive "Jenkins Is The Way" and host it at https://JenkinsIsTheWay.io .
To aggregate all these stories, we built a simple online questionnaire so that Jenkins users can submit their own experience using this leading open source automation server. With so many plugins to support building, deploying, and automating your projects, we expect to see a vast collection of stories.
We’ve already received a handful, including stories that illustrate how Jenkins Is The Way:
Add your story. Show your Jenkins pride. Get our T-shirt
Be an inspiration to the Jenkins community by sharing your Jenkins story. Just go to this link and fill out the form. We’ll ask you about your project’s goals, the technical challenges you overcame with Jenkins, and the solutions you created. It should take no more than 20-30 minutes to complete.
We’ll clean it up for clarity and publish it on https://JenkinsIsTheWay.io .
Once it’s part of our archive, we’ll send you our new 2020 Jenkins Is the Way t-shirt.
And since the more, the merrier, please share this blog post with peers and colleagues. We want to hear everyone’s stories about the clever ways Jenkins is used to automate all that we need to do.
Thanks and Acknowledgement
Special thanks to abConsulting for creating and managing the https://JenkinsIsTheWay.io site and for reviewing, editing, and publishing the submitted stories.
Thanks to the Jenkins Advocacy and Outreach SIG for their reviews and feedback.
Thanks also to CloudBees for sponsoring the "Jenkins is the Way" program.
