Tiny-EV Micro-Cars and AI Autonomous Vehicles

By Lance Eliot, the AI Trends Insider

Suppose you had available an Electrical Vehicle (EV) which was a small sized car that was incredibly inexpensive and could allow increased mobility for thousands or perhaps millions of people that otherwise did not have ready access to personal transportation? 

These so-called tiny-EV’s are quickly gaining ground and especially so in China. 

Sometimes also referred to as micro-EV’s or micro-cars, it is estimated that last year there were 1.75 million of them sold in China, which essentially doubles the number of conventional sized EV cars that were sold in China in that same year (an estimated 777,000).

There is a plethora of tiny-EV models and makers in China, numbering over 400 such automakers. I realize that those familiar with this auto industry segment might carp about calling all of those manufacturers “automakers” since the creation of tiny-EV’s is not quite the picture of what we normally consider a true automaker. Many of the firms making these tiny-EV’s are nearly working out of a garage or auto parts boneyard. The tiny-EV’s are at times outfitted with a garish array of parts and would generally fail most safety standards tests.

Speaking of safety, these tiny-EV’s are often limited as to how fast they will go. 

The typical top range speed is around 25 to 45 miles per hour. This is presumably fast enough to get you readily to where you want to go, assuming shorter distance trips, and yet not so fast that it can get you into undue trouble. There are already various kinds of Low Speed EV’s (LSEV) in the marketplace, perhaps you’ve seen them at golf courses, retirement home parks, and the like, though the LSEV’s are usually more sturdily constructed and also intended for very limited driving environments and conditions.

One of the largest criticisms of the tiny-EV’s is that they are made cheaply, and this includes tossing into the micro-car a lead-acid ultra-cheap battery. 

These no frills and dirt-cheap batteries are a potential toxic hazard. The booming growth of the tiny-EV’s market is regrettably also creating a booming number of these ecologically damaging batteries. Some are worried that in the rush toward providing the tiny-EV’s, a seemingly “good” thing for society, there is a pell-mell rush towards messing up the environment and causing some horrendous long-term health consequences.

Currently, in most jurisdictions in China, there is no requirement that the driver of a tiny-EV must have a driver’s license. You could argue that this is okay since the micro-cars are, well, like driving amusement park bumper cars, and it doesn’t take a rocket scientist to drive one. Others would say that it is a car, and as such, it ought to require having a driver’s license. It’s not simply having a piece of paper that concerns those that want a driver’s license to be required, and instead that the driver’s are many times ill-versed in driving.

 Dangers Of Tiny-EV Micro-Cars

You can obviously still do a lot of damage when driving a car that’s topping out at 25 to 45 miles per hour. 

This “low speed” is still fast enough that you can readily hit and injure or kill a pedestrian. 

You could hit and injure or kill an animal that’s on the roadway. 

You could crash into other tiny-EV’s and create injuries or deaths. 

You could crash into full-sized cars and create injuries or deaths. 

You could smash into light posts, fire hydrants, and other property.

Overall, just because it is a tiny-EV does not mean that it cannot get involved in car collisions and nor that somehow those collisions will be injury-free or damage-free simply due to the lower top speeds involved.

 While we’re discussing collisions, let’s also mention that the tiny-EV’s are typically bereft of any substantive safety equipment and related doodads. 

The odds are that if you do get into an accident, there aren’t any airbags to protect you, and the frame of the micro-car is marginally going to protect you. All told, you are pretty much driving a sardine can and anything that goes awry is going to potentially “fry” the sardine (that’s the driver and possibly a squeezed-in passenger).

There is a mounting concern that people will choose to buy a tiny-EV even though they might have been able to afford a conventional full-sized car EV. At the cost of a tiny-EV, you could probably get several of them in comparison to getting one fuller sized EV. You can easily get a tiny-EV for everyone in your family, one for you, one for each of your offspring, one for your each of your relatives, and still have money left over. If this though what the public ought to be doing?

 By encouraging the purchasing of the tiny-EV’s, it is drawing away from the conventional EV market, which might lead to the demise or at least a delay of shifting us all toward full-sized EV’s. Meanwhile, the tiny-EV’s are increasing the roadway dangers for the drivers and the driving public. Yes, they are cheap and easy, but it could be a kind of invasion that we later on realize was a mistake to let happen. Tons of tiny-EV’s and their toxic batteries, unlicensed drivers, high safety risks, and other downsides are not what everyone wants to see become prevalent on our roadways and certainly not become a dominant kind of EV. Indeed, there are some jurisdictions in China that have banned the sale and even the use of tiny-EV’s on their roads.

 On a jurisdictional matter, there is also a muddled indication of where you can drive the tiny-EV’s. In some cities in China, you can drive them not only in the roadway with other full-sized cars, but you can also drive the tiny-EV’s in the bicycle lanes.

 Imagine that you are riding your bike in a bike lane and all of a sudden a “monstrous” tiny-EV zips up to you and nearly runs you off the road. This seems like a rather dangerous place for a tiny-EV to go. You can likely guess that the drivers of the tiny-EV’s relish having legal access to use the bike lane. There you are in your tiny-EV, stuck in regular traffic and these behemoth regular-sized cars are all around you, and you suddenly dart into the bike lane to get ahead of the rest of the traffic.

 I suppose it might almost feel like being surrounded by angry bees. If you are sitting in a regular car and waiting for the traffic ahead of you to start moving, it must be somewhat disconcerting to suddenly see these tiny-EV’s racing down the bike lane and going past you. I’d bet too that the tiny-EV’s then try to merge radically back into the regular traffic, particularly if the bike lane runs out or maybe is jammed with bicycle riders.

 Let’s consider how a regular sized car needs to cope with these angry bees. You already have your hands full trying to watch for other cars of a normal size. You likely also already watch for errant pedestrians and for meandering bicycle riders. Add to your list these tiny-EV’s that can go relatively fast, meaning that at 25 to 45 miles per hour in city driving is pretty fast, and they can weave in and around the rest of the regular traffic. Can you even spot the tiny-EV that’s ahead of you? What about the one behind you? What about the one coming alongside you and in your blind spot?

 Low Cost Is Key

The typical price tag for a tiny-EV is about $1,000.

 If you were to insist that true safety features be included, it would undoubtedly jump up the price substantially. If you insisted that a driver’s license was needed, it would likely decrease by far the number of drivers that might drive a tiny-EV. If you restricted the tiny-EV’s to driving as regular cars must, meaning no more access to bike lanes, it would potentially dampen the traffic “busting” advantage of having a tiny-EV and it would become just another cog in the roadway snarl.

Essentially, making the tiny-EV’s into being proper citizens of the driving world would imply they would no longer sell. The micro-car market as we know it would likely shrivel up and collapse. There are some though that argue it would force the “auto makers” to find some other more productive means to go after the market that is apparently eager to be served. Perhaps it might spur innovation to get the costs down for adding those required safety features and perhaps the roadways could be divided up into lanes for conventional car EV’s and for the tiny-EV’s.

 Proponents of the tiny-EV’s plead to not give up on them. 

Besides the convenience for short distance trips, and the claimed traffic reducing aspects (though having lots and lots of tiny-EV’s can ultimately make traffic worse), these micro-cars are also easy to park into tight spots. You can potentially fit three tiny-EV’s into the same space needed to park a regular sized car. This reduces in theory the amount of parking spaces needed and therefore the overall set aside for car parking in a tight city locale. One might argue that the physical space normally used to park regularly sized cars could then be repurposed to be a gentle grassy park or used in some other public benefiting way.

There are some that contend the parking is actually worse off due to the tiny-EV’s. 

The tiny-EV’s are at times parked wherever the driver thinks they can get away with it. Park on the sidewalk, sure, if you can get away with it. Park in front of a fire hydrant, sure, do so if there’s room available and if you think you won’t get caught. There is also the concern that in the act of parking a tiny-EV, the driver can make rather reckless driving maneuvers. If you see a parking spot opening on the other side of the street, maybe just scoot your tiny-EV across the median, illegally, and do a quick U-turn in the roadway (illegal) and dive right into the vaunted spot.

Overall Aspects Of Tiny-EV Micro-Cars

In quick recap, there are hardly any standards regulating the tiny-EV’s. The safety features are nearly nonexistent. The use of a toxic hazard battery is a grave concern. Letting them be driven in normal traffic and into the bike lane seems to setup dangerous driving and roadway conditions. Lack of requirements of a driver’s license means that the drivers are presumably ill-prepared to properly drive the tiny-EV’s. If the tiny-EV gets into an accident, it’s bound to be an untoward result for the occupants and could cause some quite serious injury, damage, or death to other parties.

Those are many of the downsides about the tiny-EV’s.

 Let’s consider the other side of the coin, the upsides. They are inexpensive and so affordable for the masses. They are limited in their top speeds so presumably more manageable on the roadways. The tiny-EV’s allow for greater mobility and especially for those that normally might not have mobility options. Some would try to suggest they are like having a scooter, and yet more capable and perhaps even “safer” in comparison to using a scooter (of course, that’s debatable).

 Will the micro-cars grow in popularity? Many predict it definitely will. Is there a possibility of a shakeout due to the tiny-EV’s getting into killer accidents and becoming roadway pests? Many predict there definitely will be. Might it be regulated out-of-business by wanting to do the “right thing” and make them safer and more sound? Some would say that could happen, while others argue that the “auto makers” might step-up and make improvements and yet still keep many of the advantages available.

 What’s your take on the tiny-EV’s? 

Love them and keep them coming? 

Or hate them and believe they should be tossed into the junk heap? 

Time will tell.

For the power consumption aspects and EV’s, see my article: https://www.aitrends.com/ai-insider/power-consumption-vital-for-ai-self-driving-cars/

 For the advent of ridesharing, see my article: https://www.aitrends.com/selfdrivingcars/ridesharing-services-and-ai-self-driving-cars-notably-uber-in-or-uber-out/

 For the international aspects of driving cars, see my article: https://www.aitrends.com/selfdrivingcars/internationalizing-ai-self-driving-cars/

 For Personal Rapid Transit (PRT) and EV’s, see my article: https://www.aitrends.com/selfdrivingcars/personal-rapid-transit-prt-and-ai-self-driving-cars/

 AI Autonomous Vehicles And Tiny-EV Micro-Cars

 What does this have to do with AI self-driving driverless autonomous cars?

 At the Cybernetic AI Self-Driving Car Institute, we are developing AI software for self-driving cars. It is important to consider the impacts of tiny-EV’s on the advent of AI self-driving cars.

 Allow me to elaborate.

 I’d like to first clarify and introduce the notion that there are varying levels of AI self-driving cars. The topmost level is considered Level 5. A Level 5 self-driving car is one that is being driven by the AI and there is no human driver involved. For the design of Level 5 self-driving cars, the automakers are even removing the gas pedal, the brake pedal, and steering wheel, since those are contraptions used by human drivers. The Level 5 self-driving car is not being driven by a human and nor is there an expectation that a human driver will be present in the self-driving car. It’s all on the shoulders of the AI to drive the car.

 For self-driving cars less than a Level 5, there must be a human driver present in the car. The human driver is currently considered the responsible party for the acts of the car. The AI and the human driver are co-sharing the driving task. In spite of this co-sharing, the human is supposed to remain fully immersed into the driving task and be ready at all times to perform the driving task. I’ve repeatedly warned about the dangers of this co-sharing arrangement and predicted it will produce many untoward results.

 For my overall framework about AI self-driving cars, see my article: https://aitrends.com/selfdrivingcars/framework-ai-self-driving-driverless-cars-big-picture/

 For the levels of self-driving cars, see my article: https://aitrends.com/selfdrivingcars/richter-scale-levels-self-driving-cars/

 For why AI Level 5 self-driving cars are like a moonshot, see my article: https://aitrends.com/selfdrivingcars/self-driving-car-mother-ai-projects-moonshot/

 For the dangers of co-sharing the driving task, see my article: https://aitrends.com/selfdrivingcars/human-back-up-drivers-for-ai-self-driving-cars/

 Let’s focus herein on the true Level 5 self-driving car. Much of the comments apply to the less than Level 5 self-driving cars too, but the fully autonomous AI self-driving car will receive the most attention in this discussion.

 Here’s the usual steps involved in the AI driving task:

• Sensor data collection and interpretation
• Sensor fusion
• Virtual world model updating
• AI action planning
• Car controls command issuance

 Another key aspect of AI self-driving cars is that they will be driving on our roadways in the midst of human driven cars too. There are some pundits of AI self-driving cars that continually refer to a utopian world in which there are only AI self-driving cars on public roads. Currently there are about 250+ million conventional cars in the United States alone, and those cars are not going to magically disappear or become true Level 5 AI self-driving cars overnight.

 Indeed, the use of human driven cars will last for many years, likely many decades, and the advent of AI self-driving cars will occur while there are still human driven cars on the roads. This is a crucial point since this means that the AI of self-driving cars needs to be able to contend with not just other AI self-driving cars, but also contend with human driven cars. It is easy to envision a simplistic and rather unrealistic world in which all AI self-driving cars are politely interacting with each other and being civil about roadway interactions. That’s not what is going to be happening for the foreseeable future. AI self-driving cars and human driven cars will need to be able to cope with each other.

 For my article about the grand convergence that has led us to this moment in time, see: https://aitrends.com/selfdrivingcars/grand-convergence-explains-rise-self-driving-cars/

 See my article about the ethical dilemmas facing AI self-driving cars: https://aitrends.com/selfdrivingcars/ethically-ambiguous-self-driving-cars/

 For potential regulations about AI self-driving cars, see my article: https://aitrends.com/selfdrivingcars/assessing-federal-regulations-self-driving-cars-house-bill-passed/

 For my predictions about AI self-driving cars for the 2020s, 2030s, and 2040s, see my article: https://aitrends.com/selfdrivingcars/gen-z-and-the-fate-of-ai-self-driving-cars/

 Returning to the topic of tiny-EV’s, let’s consider how the advent of AI self-driving cars might be impacted.

 First, let’s consider whether or not a tiny-EV might be outfitted as an AI self-driving car.

 The odds of being able to make a tiny-EV into being a true Level 5 AI self-driving car is quite slim right now. 

At this time, the costs of the needed AI hardware and software would push the tiny-EV out of its cherished low-end pricing and instead shove the pricing way up into the stratosphere. You might as well buy a full-sized EV car if you are willing to incur the added cost for becoming an AI self-driving micro-car.

I’ve already stated in my writings and presentations that I am doubtful we’ll be able to make conventional cars into AI self-driving cars via the use of any kind of aftermarket add-ons. 

Instead, the added AI hardware and software will need to be integrated into the self-driving car. 

I’m not saying that you cannot take a conventional car and make it into a true AI self-driving car. My emphasis is that the only likely way to do so is by having the automaker and tech firm do this and not by simply selling a kit that you could go buy at your local auto parts store.

There are some that have been trying to sell such add-on kits. In my view, these kits are highly dangerous. They give the illusion that you are upgrading your car to become AI-like, but the reality is that you are merely turning your car into maybe a Level 2 or Level 3, and yet doing so in the worst of ways. 

These kits are generally something you would be wise to steer clear from using and for which absolutely they do not turn any car into a true Level 5 AI self-driving car.

For my article about the affordability of AI self-driving cars, see: https://www.aitrends.com/selfdrivingcars/affordability-of-ai-self-driving-cars/

For my article about add-on kits to make a car into an AI self-driving car, see: https://www.aitrends.com/selfdrivingcars/kits-and-ai-self-driving-cars/

 For more about conspiracy theories, see my article: https://www.aitrends.com/selfdrivingcars/conspiracy-theories-about-ai-self-driving-cars/

 For the coming issues of Level 3 self-driving cars, see my article: https://www.aitrends.com/selfdrivingcars/ai-boundaries-and-self-driving-cars-the-driving-controls-debate/

 For my article about start-ups in this space, see: https://www.aitrends.com/selfdrivingcars/how-to-best-pitch-your-startup/

 Tiny-EV Micro-Car Not The Right Size

Besides the cost barrier, there is also the aspect about the size of the AI hardware that would need to go into and onto a tiny-EV.

Imagine trying to outfit a micro-car with various radar devices, ultrasonic sensors, the LIDAR, cameras, and the like. You need to also include the various high-speed processors and memory chips. You need to add the networking communications devices, which would allow for crucial electronic communications including OTA (Over-The-Air) updating and the V2V (vehicle-to-vehicle) communications. For full-size cars and EV’s there is already some concern about the added weight, size, and impacts to the design and shape of the vehicle. For a tiny-EV, it would be many times more pronounced of an impact.

 At this time, the size and weight of those devices would severely weigh down and bloat the tiny-EV. 

You’d almost have the tail wagging the dog, in the sense that the amount of added equipment for trying to make them into an AI self-driving car might end-up making the tiny-EV into an overweight paperweight and it could barely move along. Though the sensors and other devices are admittedly increasingly getting miniaturized, please don’t expect this to happen in any miracle way such that in any near-term horizon they would be so small that they would be unnoticeably added to a tiny-EV.

I am not saying it might never happen. I don’t want to be one of those prognosticators that later on gets quoted for saying something that at the time made sense but later on looked pretty foolish. 

For example, the famous quote in Popular Mechanics magazine in 1949 that computers will be unlikely to weigh less than 1 ½ tons (which was a typical weight of the vacuum tube era mainframes), or the Ken Olsen quote in 1977 that there would be no reason for anyone to want to have a computer in their home (this was during the heyday of the minicomputer, prior to the advent of the PC).

Sure, it is possible that sometime in the future the apparatus used today for crafting an AI self-driving car will be super-inexpensive and super-tiny. The AI-related hardware could be so small and so cheap to make that it could be on the tiny-EV’s and the cost bump would be negligible and the weight difference is marginal. The AI software might be fully open sourced and not cost a dime to use. Who knows? I certainly hope that comes to fruition.

 I don’t think you should hold your breath since it is going to be a long time from now.

 For more about OTA, see my article: https://www.aitrends.com/selfdrivingcars/air-ota-updating-ai-self-driving-cars/

 For my article about 5G, see: https://www.aitrends.com/selfdrivingcars/5g-and-ai-self-driving-cars/

 For the grand convergence of high-tech, see my article: https://www.aitrends.com/selfdrivingcars/grand-convergence-explains-rise-self-driving-cars/

 Minimal On-Board And Aim For The Cloud

Another twist might be to outfit the tiny-EV’s with a minimalist set of added hardware for the self-driving car aspects and then perhaps have the rest of everything happening in the cloud.

With the emergence of 5G, perhaps you could have the AI primarily working in the cloud. 

The tiny-EV would have barebones on-board systems. Via OTA or some equivalent, the tiny-EV would be continually shoving data up to the cloud and the AI in the could would be pushing down the needed car controls commands.

I am doubtful that this is something we’ll see in the near future. 

The kind of guaranteed communication you would need is beyond today’s approaches. Presumably, if the tiny-EV’s on-board systems lost touch with the mothership system, even for a moment, it could spell disaster for the tiny-EV and its occupants and bystanders. For now, the crux of the AI self-driving car capabilities needs to be on-board of the self-driving car.

I realize you might ask what about edge computing, perhaps having computing capabilities at the side of the roadways and therefore more likely ensuring rapt communications. Yes, that’s another possibility. This use of the nearby computing would have various other risks and concerns. Not that it cannot be undertaken, but only that it is a long way off in the future too, perhaps as far away as the notion of the super-inexpensive and super-small sensors and other on-board hardware that might someday emerge.

 For the Gen Z and the future of AI self-driving cars, see my article: https://www.aitrends.com/selfdrivingcars/gen-z-and-the-fate-of-ai-self-driving-cars/

 For more about sensors and LIDAR, see my article: https://www.aitrends.com/selfdrivingcars/lidar-secret-sauce-self-driving-cars/

 For my article about edge computing, see: https://www.aitrends.com/selfdrivingcars/edge-computing-ai-self-driving-cars/

 For my article about sensors issues, see: https://www.aitrends.com/selfdrivingcars/going-blind-sensors-fail-self-driving-cars/

 Use A Federated Approach

Another perspective might be a federated approach.

Suppose we have each of the tiny-EV’s with a minimalist set of AI related hardware. Let’s also assume that there will be other nearby AI self-driving cars. This seems a reasonable assumption once AI self-driving cars become relatively prevalent. Of course, at the start of the emergence, it would not be the case and therefore this proposed federated approach would seem unlikely or premature.

 In any case, in the federated approach we divide up the chores of doing the self-driving among several self-driving cars. 

You essentially split-up the workload. Some might say this is somewhat akin to how blockchain and Distributed Ledger Technology (DLT) functions. You push around the computational aspects and exploit a distributed kind of AI approach.

During the driving effort, each of the tiny-EV’s is sharing with other nearby tiny-EV’s, and perhaps also sharing with normal sized cars that are using AI self-driving car technology. Any of the tiny-EV’s requests another nearby AI self-driving car to aid in figuring out the local surroundings and what is taking place. Via the use of V2V, plus likely the use of V2I (vehicle-to-infrastructure) electronic communications, it is conceivable that each of the AI’s can help the other out.

 This would be quite tricky and would need to include balancing the workloads. 

None of the requests for assistance can end-up in a starvation mode. Furthermore, there is a chance that the other AI’s get themselves into a swamped mode and are too overloaded. But, this is something of active research and the use of AI swarms is predicted to eventually become an essential aspect of overall AI systems deployments.

For my article about blockchain and DLT, see: https://www.aitrends.com/selfdrivingcars/blockchain-self-driving-cars-using-p2p-distributed-ledgers-bitcoin/

For my article about federated Machine Learning, see: https://www.aitrends.com/selfdrivingcars/federated-machine-learning-for-ai-self-driving-cars/

 For the role of plasticity, see my article: https://www.aitrends.com/selfdrivingcars/plasticity-in-deep-learning-dynamic-adaptations-for-ai-self-driving-cars/

 For more about AI swarm intelligence, see my article: https://www.aitrends.com/selfdrivingcars/swarm-intelligence-ai-self-driving-cars-stigmergy-boids/

 True Self-Driving Cars Coping With Tiny-EV Micro-Cars

I’ll repeat my earlier point that tiny-EV’s in the near-term have little or no chance of becoming true Level 5 AI self-driving cars. For all the reasons I’ve already mentioned, it won’t be happening any time soon, if ever.

That’s not the only angle related to AI self-driving cars.

 Here’s another perspective to consider, namely, how will AI self-driving cars cope with the tiny-EV’s? Assume for now that the tiny-EV’s are exactly as already stated in terms of being relatively unsafe and driven in a rather wild manner by unlicensed drivers.

 I had already pointed out that a human driver of a normal sized car must find these “angry bees” quite a handful to deal with. Where is that darned tiny-EV? Is it behind me, in front of me, or maybe barreling along in the bike lane and I cannot yet see it or otherwise detect it?

 We need to ask the same kinds of questions about an AI self-driving car of a normal size. Will the sensors be able to detect this zipping along tiny-EV’s? Is the AI able to deal with having them as they rapidly and with a lowered profile enter into and out of traffic wherever they seem to want to do so?

 It is already tough enough for the AI sensors to detect conventional sized cars. I realize you might say that is the sensors can detect a scooter or a motorcycle or a bike rider, shouldn’t it be able to detect the tiny-EV’s? I’d say what makes the tiny-EV an added twist is the speed factor, namely that it can go 25 to 45 miles per hour, which is likely faster than most scooters and most bike riders. In that manner, the tiny-EV is more like the maneuverability and speed of a motorcycle.

 I think we can say that the AI should be able to much of the time detect the tiny-EV’s, but without having augmented the sensors, sensor fusion, the virtual model updating, and the rest of the AI system to deal with tiny-EV’s, I would suggest that an AI self-driving car would not be as fully prepared.

 It will be important to directly and intentionally build into the AI system the capabilities of dealing with the tiny-EV’s. 

If you leave the AI to do whatever it already can do about other forms of mobile transport, I’d say there are chances of gaps or holes in what the AI ought to be doing about tiny-EV’s.

For dealing with motorcyclists, see my article: https://www.aitrends.com/selfdrivingcars/motorcyclist-entanglement-avoidance-ai-self-driving-cars/

 For pedestrians aspects, see my article: https://www.aitrends.com/selfdrivingcars/avoiding-pedestrian-roadkill-self-driving-cars/

 For my article about the aspects of cognition timing, see: https://www.aitrends.com/selfdrivingcars/cognitive-timing-for-ai-self-driving-cars/

 For my article about the fail-safe AI elements involved, see:https://www.aitrends.com/selfdrivingcars/fail-safe-ai-and-self-driving-cars/

 For safety aspects, see my article: https://www.aitrends.com/selfdrivingcars/safety-and-ai-self-driving-cars-world-safety-summit-on-autonomous-tech/

 Conclusion

 Tiny-EV’s. 

Sounds kind of quaint. 

They aren’t quaint per se. 

They are on the roadways. They present a potential danger to their occupants and bystanders. They are cars but ones without the basics of safety and furthermore vehicles that can inadvertently encourage drivers to do rather wild kinds of driving. Coupled with allowing the tiny-EV to use bike lanes, it is a potential recipe for disastrous results on the roads.

That being said, it is a societal question as to whether or not the risks/rewards are appropriate. 

If we were to try and boost the safety and other facets of the tiny-EV, it would no longer have the mass appeal and affordability that it does today. We are very unlikely to see any true AI self-driving car tiny-EV’s in the foreseeable future due to the burden of cost and size that would make the tiny-EV no longer viable. The faraway future might prove otherwise due to the ongoing and unrelenting efforts to make AI systems less expensive and smaller in size.

Meanwhile, the AI systems being devised for true Level 5 AI self-driving cars need to make sure they incorporate the nuances of the tiny-EV’s in terms of their smaller size, their agility of slipping around in traffic, their potential wildness in terms of how they are being driven, and so on. There is already a lot that we expect a proficient and true AI self-driving car to be able to do, so let’s make sure that we don’t neglect those tiny-EV’s. Angry bees are something not to be ignored. 

The AI system must have a bit of sufficient beekeeping to contend with and live in harmony with them, those tiny but powerful micro-car EV’s.

Copyright 2019 Dr. Lance Eliot 

This content is originally posted on AI Trends.

 

Ma vs Musk on AI: The Optimistic Versus the Dystopian Viewpoint

By AI Trends Staff

When you get Jack Ma and Elon Musk on the same stage debating the impact of AI, you get a study in contrasts.

Musk is CEO of Tesla and Ma is chairman of Alibaba Group Holding. The two shared a stage recently at the World Artificial Intelligence Conference in Shanghai. 

Here is a selection of the exchanges published by Bloomberg:

Regarding AI, Musk said, “People underestimate the capability of AI. They sort of think like it’s a smart human. (But it’s going to be) much smarter than the smartest human you will ever know.”

Ma said, “I never in my life say human beings will be controlled by machines, it’s impossible…Human beings can never create another thing that is smarter than human beings.”

Musk said, “I very much disagree with that.”

In another contrasting exchange, Ma said, “I’m quite optimistic and I don’t think artificial intelligence is a threat.” Musk’s retort included the phrase, “famous last words.”

This debate epitomizes the clash between the “narrow” AI embedded in GPS systems and Amazon recommendations, and the goal of artificial general intelligence, a self-teaching system able to outperform humans across a range of disciplines. This is known as the “singularity,’ which some scientists believe is 30 years away, if it is ever achieved. 

A recent account in Smithsonian quoted luminaries on many sides of this debate. In the area of employment, the prediction is that AI will be able to perform many jobs now done by humans,  including drivers and insurance adjusters. The optimistic view is that this will lead to government paying unemployed citizens a universal basic income, freeing them to pursue their dreams. The pessimistic view is this will create great wealth inequality and possibly failed nations around the globe.   

Ma was optimistic about the impact of AI on employment, however his views have shifted, noted an account of the Musk-Ma debate in Forbes. He said AI has the potential to cut the time spent in the office to 12 hours per week, freeing up time for other pursuits. “For the next 10, 20 years, every human being, country, government should focus on reforming the education system, making sure our kids can find a job, a job that only requires three days a week, four hours a day,” Ma said.

That was a changed sentiment from Ma’s statement in April this year in which he endorsed the idea of working from 9 am to 9 pm six days a week, like employees of many startups and tech giants in China.  That schedule is nicknamed “996.” Ma had said in a blog post on Weibo, the Chinese social media site, “996 is not a problem.”

Read the source posts in Bloomberg, Smithsonian and Forbes.

YouTube Using AI to Help Remove Video Deemed Offensive; Meanwhile Recommendation Engine is Challenged

By AI Trends Staff

You Tube needs to employ AI to help process the 300 hours of video uploaded to the platform every minute by its users. This processing includes removing video deemed inappropriate by YouTube’s standards. 

Some 8.3 million videos were removed from YouTube in the first quarter, 76 percent of those identified and flagged by AI automatically, according to an account in Forbes. Of those, more than 70 percent were never viewed by users. While the AI system is able to review more content than humans, full-time human specialists work with the AI, which of course is not foolproof.

YouTube’s “number one priority” is to prevent harmful content from seeing the light of day via YouTube, said Cecile Frot-Coutaz, head of the EMEA region for YouTube, based in London. AI and machine learning has advanced the company’s ability to identify objectionable content, with performance improving from eight percent “pre-AI” of banned video being removed before 10 views, to more than 50 percent being removed.

An important metric used by YouTube’s algorithms is video “watch time,” valued by advertisers. However, the metric tends to amplify videos with outlandish content, and the more people watch it, the more highly it is recommended, notes Guillaume Chaslot, a former Google employee and founder of AlgoTransparency, a firm that encourages greater transparency in algorithms. Chaslot gave an address at the recent DisinfoLab Conference.

Chaslot is not a fan of the YouTube recommendation engine. “If the AI is well-tuned, it can help you get what you want. But the problem is that the AI isn’t built to help you get what you want — it’s built to get you addicted to YouTube. Recommendations were designed to waste your time,” he is quoted as saying an address at the recent DisinfoLab Conference, in an account from TheNextWeb.

AlgoTransparency built a program to analyze what videos YouTube is recommending each day. The website states, “The algorithm is responsible for more than 700,000,000 hours of watch time every day, and it is not fully understood, even by the people who built it.” From its base of 1,000 channels tracks, it shows how many recommended the top videos on YouTube each day.

In general, Chaslot has seen that the close a video gets to the edge of what is acceptable under YouTube’s policy, the moe engagement the video will get. “We’ve got to recognize that YouTube recommendations are toxic and pervert civic discussion,” he said. “Right now the incentive is to create this type of borderline content that’s very engaging, but not forbidden,” 

Google, which bought YouTube in 2006, and YouTube have challenged Chaslot’s methodology. He said his requests to probe the flaws in cooperation with YouTube have gone unanswered. Until Google becomes more transparent about how it recommends videos, the engine will not be better understood.

Going over the edge of propriety too many times on its homepage spurred YouTube to recently employ a “trashy video classifier” for its homepage, according to an account in Bloomberg. Early experience has shown the fix is helping keep many inappropriate clips off the home page. The “watch time” on YouTube’s homepage has grown 10x in the past three years, Google marketers said recently.

Read the source posts in Forbes,  TheNextWeb and  Bloomberg.

Multi-Cloud Strategy a Fit for Large Enterprises; Wells Fargo Finding It Helps to Have Flexibility to Move Data

By AI Trends Staff 

Enterprises are engaging in a multi-cloud strategy to distribute their AI across the different capabilities of the cloud providers. 

Wells Fargo for instance has made multi-cloud a key part of its strategy. “You’re going to pick on the provider who has differentiated themselves and at that point it becomes — it really has to become part of your strategy to be agile to move that workload across from one provider to the other,” said Mike Telang, executive vice president and head of enterprise architecture at Wells Fargo,at the recent VMWorld 2019 in San Francisco, quoted in ZDNet.

The cloud strategy at Wells Fargo is a lot about where the data goes. “When we think of workload we typically think about applications, but for us, I think the value will be in the ability to move our data from one to the other without leaving that data behind,” Telang said. “So our cloud strategy, when we think of multi-cloud, we’re really trying to kind of simplify it by saying, ‘What’s our strategy around SaaS, what’s our strategy around PaaS, what’s our strategy around IaaS’ — where we see the big data go, so we’re looking at it that way.”

Wells Fargo has applied AI and machine learning to help protect against fraud and money laundering. At the same time, regulators want to know how decisions are made about providing credit or loans, to try to ensure all customers are treated fairly. Telang called this a “double-edged sword” for banks.

Following the Data in a Multi-Cloud World

In today’s multi-cloud world for large enterprises, AI tools are able to access and manage an endless amount of data. Companies that have applied AI tools to cloud silos of data may have to be retooled to fit a multi-cloud infrastructure with AI at its core, suggested Subram Natarajan, chief technology officer of IBM India, in a recent article published in Tech2/First Point.

While a one-cloud-fits-all approach might have looked appealing in the early days of cloud computing, in reality business functions and IT arms of organizations picked and chose a mix of public and private clouds for their data centers, depending on the business need. The hybrid multi-cloud strategy has proven successful for many companies, for cost effectiveness and the ability to develop and deploy applications quickly. Also, the strategy can cater to the unique requirements of business units at different stages of maturity. Concerns about security and data governance can be addressed at a business unit level as well as company-wide.

The more relevant data available to AI systems the better, so multi-cloud is consistent with AI rollouts. “AI in a multi-cloud infrastructure helps developers, who can leverage the building blocks of technology that exist within respective cloud environments,” Natarajan said. 

Research from Ovum show that while 20 percent of business processes have moved to the cloud, 80 percent of mission-critical workloads are still running on-premise, due to performance and regulatory requirements. Business managers and IT professionals are finding that moving data and applications across different on-premise and cloud computing infrastructures is difficult. The ideal is for data to flow seamlessly and be able to interact across clouds to realize greater value. 

Read the source articles in  ZDNet and  Tech2/First Point.

A Guide to Developer, Deep Dive, and Apache Spark Tutorial Talks at Spark + AI Summit, Europe

You might have heard the famous saying, “Why software is eating the world.” But if software is eating the world, you may ask, where does software come from?

Naturally, Developers! Some software developers advocate that the “Developers are eating the world.” A research report by Stripe indicates that “developers have the ability to raise global GDP by $3 trillion over the next 10 years.” Perhaps so.

But their dominance at data-driven companies to produce data products affecting revenue is uncontested; their contributions to open-source projects on GitHub is unmatched; their contribution and presence at technical conferences are notable and influential; and their commitment to open-source community meetups is enduring.

source: https://open.spotify.com/show/6ugAoqTgAaNmd9UwybEsjR?si=SqTj2NU2Qyy3f2oTOV-wLw

In this blog, we highlight selected sessions by developers for developers that speak of their endeavors in combining the immense value of data and machine learning across sessions focused on Developer, Deep Dives, and Tutorials.

Developer and Deep Dives

Naturally, let’s start with the Developer track. Messrs Martin Junghanns and Sören Reichardt of Neo4J will share a new contribution to Apache Spark 3.0: Extending Spark Graph for the Enterprise with Morpheus and Neo4j. A new module for graphs in Spark, this session introduces how to transform data into Property Graphs using Morpheus and Cypher APIs.

Related to Graphs in Apache Spark, Dr. Victor Lee and Songting Chen of TigerGraph will compare three options for using graphs in Spark: GraphX, Cypher for Apache Spark, and TigerGraph. Don’t miss his talk, Assessing Graph Solutions for Apache Spark. 

Both contributions from the community enhance and extend Spark with graphing capabilities.

Which brings us to Spark’s extensibility. Among many features that attract developers to Spark, one is its extensibility with new language bindings, libraries or extension of its components. Messrs Terry Kim and Rahul Potharaju of Microsoft will explain how they extended Spark to include a new .NET bindings in their talk: .NET bindings for Apache Spark.

Another session that shows Spark extensibility is a deep dive and live coding session, Extending Spark SQL 2.4 with New Data Sources. Jacek Laskowski, an independent consultant and author of Apache Spark Internals, will show in a live coding session how developers can extend Spark SQL with new or customized data sources.

The new open-source project Delta Lake extends Apache Spark to add ACID reliability to Data Lakes. In the talk, Databricks Delta Lake and Its Benefits, Nitin Raj and Nagaraj Sengodan of Cognizant Worldwide Limited will share how Delta Lake APIs are completely compatible with Apache Spark and how its transactions capabilities bring reliability to Data Lakes.

For software developers interested in internals and optimization of Apache Spark, a few sessions standout: First, Apache Spark’s Built-in File Sources in Depth, from Databricks Spark committer Gengliang Wang. In Spark 3.0, all data sources are reimplemented using Data Source API v2. This session will explain what those are and how to optimally use them.

Second, Luca Canali, from CERN, will explain performance troubleshooting of distributed data processing and improvements in Apache Spark 3.0 in his talk, Performance Troubleshooting Using Apache Spark Metrics. 

Third, Spark tuning and optimization require knowledge of what configurations to tweak for optimal resource utilization. Four sessions elaborate on what–why-how of Spark tuning: Apache Spark Core – Practical Optimization (Daniel Tomes of Databricks); Using Production Profiles to Guide Optimizations (Adam Barth of Facebook); The Parquet Format and Performance Optimization Opportunities (Boudewijn Braams of Databricks); and Internals of Speeding up PySpark with Arrow  (Ruben Berenguel, big data consultant)

MLflow, Delta Lake, Koalas, and Morpheus Tutorials

First introduced as dedicated 90-minute hands-on tutorial at Spark + AI Summit this year in San Francisco, tutorials had tremendous success in attendance and technical content, so we want to make this part of the summit in Amsterdam too. Here are few tutorials that are worth attending:

• Graph Features in Apache Spark 3.0: Integrating Graph Querying and Algorithms in Spark Graph
• Cosmos DB Real-time Advanced Analytics Workshop
• Koalas: pandas on Apache Spark
• Managing Machine Learning Life Cycle Management with MLflow
• Building Reliable Data Pipelines with Delta Lake

What’s Next

You can also peruse and pick sessions from the schedule. If you have not registered for the summit, use “Jules20,” a 20% discount code. In the next blog, we will share our picks from sessions related to Data Science, Deep Learning, Machine Learning, and AI Use Case tracks.

 

--

Try Databricks for free. Get started today.

The post A Guide to Developer, Deep Dive, and Apache Spark Tutorial Talks at Spark + AI Summit, Europe appeared first on Databricks.

Design Decisions for the First Embedded Analytics Open-Source Framework

Top Redis Use Cases by Core Data Structure Types

Redis, short for Remote Dictionary Server, is a BSD-licensed, open-source in-memory key-value data structure store written in C language by Salvatore Sanfillipo and was first released on May 10, 2009. Depending on how it is configured, Redis can act like a database, a cache or a message broker. It’s important to note that Redis is a NoSQL database system. This implies that unlike SQL (Structured Query Language) driven database systems like MySQL, PostgreSQL, and Oracle, Redis does not store data in well-defined database schemas which constitute tables, rows, and columns. Instead, Redis stores data in data structures which makes it very flexible to use. In this blog, we outline the top Redis use cases by the different core data structure types.

Data Structures in Redis

Let’s have a look at some of the data types that Redis supports. In Redis, we have strings, lists, sets, sorted sets, and hashes, which we are going to cover in this article. Additionally, we have other data types such as bitmaps, hyperloglogs and geospatial indexes with radius queries and streams. While there are some Redis GUI tools written by the Redis community, the command line is by far the most important client, unlike popular SQL databases users which often prefer ...


Read More on Datafloq

UPI transactions top 900 million in August

Newly released NPCI data show that in August, a total of 918 million transactions worth Rs 1.54 lakh crore were processed on the interoperable digital payments channel;

Commvault to acquire Hedvig for Rs 1,621 crore

Hedvig, founded in 2012 by entrepreneur Avinash Lakshman, has raised up to Rs 375 crore ($52 million) till date, with its last Series-C round being covered by HP Enterprise in 2017.

How the Internet of Things (IoT) Can be Used to Build Smart Cities?

The world’s population is increasing at a tremendous pace. As per recent figures, almost 3.9 billion people are living in cities. By 2050, it is estimated that almost two-thirds of the world population will shift to urban areas. The rapid increase of the population density inside urban environments, infrastructures, and services has been needed to supply the requirements of the citizens. So, cities need to be smart, if only to survive as platforms that allow economic, social and environmental safety.

A smart city is the one that uses information and communications technologies (ICT) to make the city services and monitoring more aware, interactive and competent. Smartness of a city is driven and enabled technologically by the growing Internet of Things (IoT)- a radical evolution of the current Internet into a global network of interconnected objects that not only gathers information from the environments (sensing) and interacts with the physical world, but also uses existing Internet standards to provide services for information transfer, analytics, and applications.

IoT Technologies for Smart Cities

The IoT is a broadband network that employs standard communication protocols, whereas the Internet would be its merging point. The major idea of the IoT is the widespread existence of objects which are ...


Read More on Datafloq

The why and the how of enterprise AI

Scheduled for October 2, 2019, 11 am to 12:00 pm EDT

Register Today!

In this webinar we will address two major questions facing organizations that are using or planning to use AI and machine learning:

• What are the use cases and business drivers behind enterprise investment now and in the future?
• What are the changes to infrastructure that are happening – and need to happen – to facilitate greater adoption of AI & machine learning at scale?

Even though we’re only at the beginning of the enterprise AI journey, enterprise adoption is quickening and deepening, especially in industries such as financial services, healthcare, manufacturing and retail. In this webinar Nick will show evidence of specific use cases in those industries to which AI and machine learning is being applied. He will also reveal the barriers organizations face to delivering AI at scale in terms of the infrastructure they have now and the types of infrastructure investments they may want to make in the future.

Nick Patience, Co-founder & Research VP, AI Applications & Platforms, 451 Research
Nick leads 451’s AI and machine learning research – an area he has been researching since co-founding 451 Research in 2000. He works across the entire 451 Research team to uncover and understand use cases for machine learning to advise 451’s end user, vendor, service provider and investor clients. He also steers the company’s coverage of business applications, including Customer Experience and Commerce and Workforce Productivity. He re-joined 451 in 2015 after almost three years running product marketing at machine learning-driven eDiscovery and search company Recommind (now part of OpenText). He has held various senior management roles at 451 in both in New York and London since 1999. Prior to starting 451 Research, Nick was a financial and technology journalist with ComputerWire (now part of Datamonitor) in London and New York. Nick has an MSc in Computing Science from the University of London, and a BA in Philosophy and Music from Middlesex University.

Register Today!

Exciting Keynotes at Spark + AI Summit Europe 2019

Spark + AI Summit is the premier global event for the data and machine learning community to discuss the latest advances in open-source technologies such as Apache Spark™, Delta Lake, MLflow, Koalas and TensorFlow as well as best practices for deploying AI in the real world.

In addition to over 100 exciting breakout sessions, this year’s Spark + AI Summit in Amsterdam 15-17 October will feature keynotes from some of the leading thinkers and innovators in AI. In this blog, we’ll highlight a few of the esteemed keynote speakers, including the original creators of Spark, Delta Lake, and researchers from leading academic institutions.

Keynotes You Won’t Want to Miss!

Developing ML Algorithms to Image Black Holes

Katie Bouman Assistant Professor Computing and Mathematical Sciences, Caltech

We’re pleased to announce Katie Bouman’s keynote address at this year’s Spark + AI Summit Europe. Katie was a postdoctoral fellow in the Harvard-Smithsonian Center for Astrophysics, and she received her Ph.D. from MIT’s Computer Science and Artificial Intelligence Laboratory in EECS. Currently, she specializes in using emerging computational methods to push the boundaries of imaging. In her keynote address, Katie will discuss how she developed an ML algorithm to capture the first-ever picture of a black hole.

Building AI to Beat Professional Starcraft Players

Oriol Vinyals Principal Scientist Google DeepMind

We’re excited to welcome Oriol Vinyals to this year’s Spark + AI Summit Europe. Oriol holds a Ph.D. in EECS from UC, Berkeley, and he received the 2016 MIT TR35 innovator award. In his keynote address, AlphaStar: Mastering the Real-Time Strategy Game StarCraft II with AI, Oriol will discuss AlphaStar, the first AI program to defeat a top professional StarCraft player under professional match conditions. Games are critically important in testing and evaluating AI systems, and StarCraft has emerged as a “grand challenge” for AI research.

Insights from Matei Zaharia, the Original Creator of Apache Spark and MLflow

Matei Zaharia CTO, Databricks and Assistant Professor of Computer Science, Stanford

Matei Zaharia will be joining us at this year’s summit to share his latest insights on streamlining the end-to-end machine learning lifecycle. Matei initiated the Apache Spark project in 2009, while working on his Ph.D. at UC, Berkeley, and he has also worked on datacenter systems, co-creating the Apache Mesos project and contributing as a committer on Apache Hadoop. He is currently an Assistant Professor of Computer Science at Stanford University and Chief Technologist at Databricks, where he heads the MLflow development effort. Matei’s research work has been recognized by ACM, with its 2014 Doctoral Dissertation Award for the best computer science Ph.D. dissertation, along with receiving an NSF CAREER Award and several best paper awards.

Democratizing Machine Learning: Perspectives from a scikit-learn Creator

Gaël Varoquaux Brain Imaging Research Inria

Gaël Varoquaux is an Inria faculty researcher, specializing in data science and brain imaging. He holds a joint position at Inria (French Computer Science National research) and the Neurospin Brain Research Institute. Gaël’s research focuses on using data and machine learning for scientific inference, applying it to brain-imaging data to understand cognition, and developing tools that simplify the use of machine learning for non-specialists. During his keynote, Gaël will share insights from his research on how to democratize machine learning for the masses. He has long dreamed of making bleeding-edge data processing available to new fields, and he is working on a master plan to build easy-to-use open-source software in Python. He is a core developer of scikit-learn, joblib, Mayavi, and nilearn, and a nominated member of the PSF.

Delta Lake Open Source Community Momentum

Michael Armbrust Principal Software Engineer Databricks

Michael Armbrust is committer and PMC member of Apache Spark and the original creator of Spark SQL and Delta Lake. He currently leads the team at Databricks that designed and built Structured Streaming and Delta Lake. During his keynote, he will share insights from leading these projects. He received his PhD from UC Berkeley in 2013, and was advised by Michael Franklin, David Patterson, and Armando Fox. His thesis focused on building systems that allow developers to rapidly build scalable interactive applications, and specifically defined the notion of scale independence. His interests broadly include distributed systems, large-scale structured storage and query optimization.

Find out more

Learn more about Spark+AI Summit, including descriptions of breakout sessions and a complete schedule.

Related Resources

• Spark + AI Summit 2019
• AI Trends that Paved the Way in 2018
• The Biggest AI Trends from 2019

 

--

Try Databricks for free. Get started today.

The post Exciting Keynotes at Spark + AI Summit Europe 2019 appeared first on Databricks.

What Tools Should a Good Analyst have in Their Toolbox? (part 2 = people skills)

Continuing Martin Squires series on the tools that should be in a good analyst toolbox. In part 2, Martin moves on to focus on the people skills that are needed.

I use the term people skills for a couple of reasons. Firstly, there has been some online outrage at the overuse of the term Softer Skills. Secondly, because the skills that Martin lists are very people-centric. That alchemy of attitude and interpersonal interactions that can make all the difference.

So, back to Martin to share what he has found is important amongst the many analysts he has the lead. It may be many years since Martin has been a “hands-on” analyst, but like me, he has had many years to spot what makes the difference between analysts who flourish or fail. It is rarely technical skills.

From analyst toolbox to golf club bag

In theory, my list of technical tools should give you a pretty good club bag to get around most courses. But, it does leave me with one thought. A list of tools and resources are all well and good, but you could give me Tiger Woods clubs and I’d still be putting the ball in the next field half the time.

What about the other skills you need as an analyst? The toolbox definitely needs to contain some “softer skills” too. I call them the ...


Read More on Datafloq

Drone startup Aarav Unmanned Systems get the license to fly

The company works with large mining, infrastructure, power and survey companies, including with several state-run agencies

Transform Your AWS Data Lake using Databricks Delta and the AWS Glue Data Catalog Service

In this blog post we will explore how to reliably and efficiently transform your AWS Data Lake into a Delta Lake seamlessly using the AWS Glue Data Catalog service. The AWS Glue service is an Apache compatible Hive serverless metastore which allows you to easily share table metadata across AWS services, applications, or AWS accounts.

This provides several concrete benefits:

• Simplifies manageability by using the same AWS Glue catalog across multiple Databricks workspaces.
• Simplifies integrated security by using Identity and Access Management Credential Passthrough for metadata in AWS Glue. Refer to the Databricks blog introducing Databricks AWS IAM Credential Passthrough for a detailed explanation.
• Provides easier access to metadata across the Amazon services and access to data catalogued in AWS Glue.

Databricks Delta Lake Integration with the AWS Core Services

This reference implementation illustrates the uniquely positioned Databricks Delta Lake integration with the AWS core services helping you solve your most complex Data Lake challenges.

Overview

What is Delta Lake?

Delta Lake is an open source storage layer that brings reliability to data lakes. Delta Lake provides ACID transactions, scalable metadata handling, and unifies streaming and batch data processing. Delta Lake runs on top of your existing data lake and is fully compatible with Apache Spark APIs.

Download this ebook to gain an understanding of the key data reliability challenges typical facing data lakes and how Delta Lake helps address those challenges.

Databricks recently open-sourced Delta Lake at the 2019 Spark Summit. You can learn more about Delta Lake at delta.io.

Presto and Amazon Athena compatibility support for Delta Lake

As of Databricks runtime 5.5, you can now query Delta Lake tables from Presto and Amazon Athena. When an external table is defined in the Hive metastore using manifest files, Presto and Amazon Athena use the list of files in the manifest file rather than finding the files by directory listing. These tables can be queried just like tables with data stored in formats like Parquet.

Step 1. How to configure a Databricks cluster to access your AWS Glue Catalog

First, you must launch the Databricks computation cluster with the necessary AWS Glue Catalog IAM role. The IAM role and policy requirements are clearly outlined in a step-by-step manner in the Databricks AWS Glue as Metastore documentation.

For the purpose of this blog, I’ve created an AWS IAM role called Field_Glue_Role which also has delegated access to my S3 bucket. I’m attaching the role to my cluster configuration, as depicted in Figure 1.

Figure 1.

 

Next, the Spark Configuration properties of the cluster configuration must be set prior to the cluster startup as shown in Figure 2.

Figure 2. Updating the Databricks Cluster Spark Configuration properties

Step 2. Setting up the AWS Glue database using a Databricks notebook

Before creating an AWS Glue database let’s attach the cluster to your notebook, created in the previous step, and test your setup issuing the following command:

Then validate that the same list of databases is displayed using the AWS Glue console and list the databases.

We are now ready to create a new AWS Glue database directly from our notebook as follows:

And verify that the new AWS Glue database has been created successfully by re-issuing the SHOW DATABASES. The AWS Glue database can also be viewed via the data pane.

Step 3. Create a Delta Lake table and manifest file using the same metastore

Now, let’s create and catalog our table directly from the notebook into the AWS Glue Data Catalog. Refer to how Populating the AWS Glue data catalog for creating and cataloging tables using crawlers.

I’m using the movie recommendation site MovieLens dataset which is comprised of movie ratings. I first created a DataFrame with this python code:

and then register the DataFrame as a temporary table to access it using SQL as follows:

Now let’s create a Delta Lake table using SQL and the temporary table created in the previous step:

Note: It’s very easy to create a Delta Lake table as described in this Delta Lake Delta Lake Quickstart Guide.

We can now generate the manifest file required by Amazon Athena using the following steps.

1.     Generate manifests by running this Scala method. Remember to prefix the cell with %scala if you have created a python, SQL or R notebook.

2.     Create a table in the Hive metastore connected to Athena using the special format SymlinkTextInputFormat and the manifest file location:

In the above sample code, notice that the manifest file is created in the s3a://aws-airlifts/movies_delta/_symlink_format_manifest/ file location.

Step 4. Query the Delta Lake table using Amazon Athena

Athena is a serverless service that does not need any infrastructure to manage and maintain. Therefore, you can query the Delta table without the need of a Databricks cluster running.

From the Amazon Athena console, select your database, then preview the table as follows:

Conclusion

With the support of AWS Glue we have introduced a powerful serverless metastore strategy for all enterprises using the AWS ecosystem. Furthermore, we are elevating the reliability of your Data Lake with Delta Lake and provide seamless as well as serverless data access for your enterprise by integrating with Amazon Athena.

You can now safely enable your analysts, data engineers, and data scientists to use the Databricks Unified Analytics Platform to power your Data Lake strategy on AWS.

 

Related Resources:

• Open Sourcing Delta Lake
• eBook: Building Reliable Data Lakes at Scale with Delta Lake
• Delta Lake Quickstart
• Databricks Runtime

Try it!

--

Try Databricks for free. Get started today.

The post Transform Your AWS Data Lake using Databricks Delta and the AWS Glue Data Catalog Service appeared first on Databricks.

What Tools Should a Good Analyst have in Their Toolbox? (part 1 = technical)

Continuing our theme of tools for analytics teams, what tools should analysts have in their toolbox? It’s a broad question and one with diverging views. So, I am delighted to welcome back a guest blogger who doesn’t shy away from controversy.

Martin Squires is a very experienced Analytics leader, whom I’ve previously interviewed in our audio series. He has also posted before on why he disagrees with the use of Business Partners (a view I countered here).

I hope this post might be the start of a series of perspectives. But, for now, over to Martin to share his wisdom on what should be in your toolbox…

Getting caught up in comparison

With the plethora of new kit that seems to come out every year (or even month), it’s easy to get lost in the depths of debates about R vs Python, Azure vs AWS etc etc. When asked about my favourite tools it reminded me that I don’t get the pleasure to be “hands-on” for a living anymore.

But what I do have to do is to try and make sure teams I lead have the right tools, competency frameworks and development plans. So I thought I’d take it back a step or two and try and look at what I’d still want to learn if I was setting out on the development path today. What I’d recommend for ...


Read More on Datafloq

Maximizing Human Life Spans and AI Autonomous Cars As A Fountain Of Youth

(Ed. Note: For reader’s interested in Dr. Eliot’s ongoing business analyses about the advent of self-driving cars, see his online Forbes column: https://forbes.com/sites/lanceeliot/)

By Lance Eliot, the AI Trends Insider

What is the secret to achieving old age?

Jeanne Calment, having lived to the age of 122, had attributed her longevity to her diet which was rich in olive oil.

Sarah DeRemer Knauss, an American supercentenarian, lived 119 years and credited her longevity to the aspect that she didn’t let things in life upset her. Indeed, when she first learned that she had become verified as longest known living person in modern history (at that point in time), she apparently said “so what?”

Or, you might find of interest the case of Susannah Jones, she happily consumed four strips of bacon for breakfast each morning, which was included with her scrambled eggs and grits, and was known to eat bacon throughout each day – she lived 116 years.

Does this mean that if you are desirous of reaching a ripe old age that you should rush out to buy lots of olive oil and bacon, along with adjusting your perspective about life to keep from getting upset?

Well, maybe. I can’t say for sure that this won’t help you, but nor can we say with any certainty that it will help you to make it into your hundreds.

One acrimonious debate about old age is whether you are born with the ability to reach it or whether it is your environment that can produce it.

Some say that it has to do with DNA. Your DNA either has some kind of longevity gene or it does not. If you weren’t born with it, you are out of luck in terms of trying to reach the topmost ages. Sorry to say. Of course, merely being born with the proclivity doesn’t guarantee it will bear out. You could perish in an earthquake, get hit by a car, or be involved in a war and die that way.

In this nature versus nurture debate, some would argue that your environment is the primary influencer for successfully reaching old age.

If you live in a place that provides a suitable climate, if you live nearby those that can help care for you when you get older, if you have medical assistance that can apply the latest life extending care, under these conditions you have a chance of achieving older age. Someone that might have a perfectly nature-designed old-age DNA can be readily wiped out sooner by living in a place and time that does not foster living to an older age.

Maybe both nature and nurture intertwine such that we cannot separate one factor from the other.

Perhaps the ultimate environment for aiding aging can keep anyone going, regardless if their genes were suitable per se. The person with super aging DNA maybe can tolerate an environment not quite as suitable and still make it. I’d guess this debate will continue for quite some time to come.

How Old Can We Become

Let’s shift the debate to another equally interesting question, namely whether there is a limit to how old someone can become?

For this question, I’m sure that most of us would say that yes, there has to be a limit. It seems unimaginable that you could just keep living and living and living.

Wouldn’t the human body just plain wear out?

We’ve seen many science fiction movies where they take someone’s brain and place it into another human’s body, a younger body, in order to achieve a kind of immortality.

A study published in Science magazine postulated that maybe we can keep doing things in our environment that can extend old age (see the article on mortality rates: https://science.sciencemag.org/content/360/6396/1459).

Perhaps there is no limit per se.

Upon analyzing various mortality rates, they suggested that once you reach the age of 105 (yes, I think you will!), the mathematically imputed probability of death seems to stop increasing.

One interpretation is that we have not yet reached a limit and we have thus yet to know what the limit is.

This does not necessarily mean that life is going to be limitless in aging, but just that we haven’t found the end point as yet.

Others that have studied aging find this to be a bit off-target in terms of how the study was conducted and the kinds of interpretations to be made of it.

First, trying to assert that there’s no upper limit seems quite speculative and not really the spirit of the data that was collected and assessed.

Second, if suppose that one person can live to the age of 140, would that be construed that we all have a chance of doing so?

In other words, the statistical anomaly of someone out of the billions of people on this planet that happens to make it to some incredible older age should not be falsely used to suggest we all can, or that even many of us can, or even that a few can. It might be a lighting strike kind of occurrence.

However this old age debate ends-up faring, the general rule-of-thumb seems to be that for most of us, we can live to a ripe old age by eating right, watching our health, taking less risks, keep our bodies in shape, and keep our minds in shape.

This is not the secret formula to get you to the hundreds, and instead the traditional advice about how to keep going to some kind of older age.

Plus, as per the famous quote by Theodore Roosevelt, old age is like anything else, to make a success of it, you’ve got to start young. Presumably, if when younger you eat poorly, don’t watch your health, take high risks, don’t keep your body in shape, and don’t keep your mind sharp, trying to suddenly change your ways at a later age might be too late. The damage was already done, some say.

AI Autonomous Cars And Maximizing Human Life Spans

What does this have to do with AI self-driving driverless autonomous cars?

At the Cybernetic AI Self-Driving Car Institute, we are developing AI systems for self-driving cars and also keenly interested in how self-driving cars will be used by society.

Here’s a thought provoking assertion: AI self-driving cars will help to maximize human life spans.

I’ve debated this topic at some industry conferences and thought you’d like to know about it.

There are already assertions that AI self-driving cars will reduce the number of car related deaths, which is considered one of the largest benefits to society for the advent of self-driving cars.

I agree that someday it is likely that AI self-driving cars will reduce the number of car related deaths, but I also claim that it is many years into the future and that for the foreseeable future it won’t materially impact the number of car related deaths. Indeed, I argue that this whole idea of “zero fatalities” is a gimmick and misleading or stated by those that are perhaps misinformed on the matter.

See my article about zero fatalities is zero chance: https://aitrends.com/selfdrivingcars/self-driving-cars-zero-fatalities-zero-chance/

Even if the advent of AI self-driving cars eliminated all car related deaths, you need to realize that the number of car related deaths per year in the United States is about 40,000.

There are about 325 million people in the United States.

As such, though every life is precious, the saving of 40,000 lives out of a population of 325 million is important but not something that will cure all deaths from happening.

There are an estimated 650,000 deaths each year in the U.S. due to heart disease, and another 600,000 deaths due to cancer. In theory, if we were only looking at number of deaths as a metric, we would say that we should take all the money spent toward AI self-driving cars and put it toward curing heart disease and cancer, since that has a much higher death rate than car related deaths.

The point here is that the AI self-driving car emergence will not presumably alter the likelihood of achieving older age by the act of reducing or eliminating deaths in the population.

That’s not going to move the needle on the old age achievement scale (though, allow me to emphasize that each life lost due to a car accident is a tragedy).

Mobility As A Factor In Longevity

What then might the AI self-driving car be able to do to advance our ages?

One aspect that is touted about AI self-driving cars is that it will increase the mobility of humans.

There are some that say we are going to become a mobility-as-an-economy type of society. With the access to 24×7 car transportation and an electronic chauffeur that will drive you wherever you want to go, it will mean that people today that aren’t readily mobile can become mobile. Kids that can’t drive today will be able to use an AI self-driving car to get them to school or to the playground or wherever they need to go.

The elderly that no longer are licensed to drive will be able to get out of their homes and no longer be homebound, doing so by making use of AI self-driving cars.

See my article about how AI self-driving cars will impact the elderly: https://aitrends.com/ethics-and-social-issues/elderly-boon-bust-self-driving-cars/

So, we can make the claim that via the use of more prevalent mobility, it could allow those that are older to be able to more readily visit with say medical advisers and ensure that their healthcare is being taken care of.

Need a trip to the local hospital? In today’s terms, it might be logistically prohibitive for the homebound elder to make such a trip. In contrast, presumably with ease they will be able to call forth an AI self-driving car that can give them a lift to the nearby medical care facility.

Access And Frequency Of Healthcare

Healthcare can also more readily come to them, including having clinicians that go around in AI self-driving cars and can visit with those that need medical assistance.

If you are willing to believe that having timely medical care is an important factor in achieving and maintaining older age, the AI self-driving car can be a catalyst for that to occur.

Boosting The Spirit And Reduce Isolation

Another case of how an AI self-driving car might contribute to the aging process in terms of prolonging life might be due to increased access to other humans and presumably gaining greater mental stimulation and joy in life.

Want to visit your grandchildren?

Rather than having to arrange for some convoluted logistics, you just get the AI self-driving car to take you to them.

Again, the reduced friction in mobility, some would say it is friction-less (I think that’s a tad over-the-top), allows for trying to keep both body and mind in shape.

Some say that isolation tends to lead to early deaths.

AI self-driving cars have the potential for increasing socialization and reducing isolation.

This is achieved by the ease of mobility. In addition, while in an AI self-driving car, it is predicted that AI self-driving cars will have all sorts of electronic communication capabilities, and during a journey you can be doing all kinds of Skype-like communication with others. Thus, even if in an AI self-driving car and all alone in doing so, you can actually be interacting with others during a driving trip.

Physical Fitness As A Factor

Another factor might be physical fitness.

If you are at home and isolated, you might not be inspired to do physical fitness.

Admittedly there are more and more in-the-home treadmills and bikes that will allow you to virtually interact with others across the globe, but this still doesn’t seem to be as meaningful and motivating as doing so in-person. With an AI self-driving car, you could readily get to some location whereby physical fitness with others is able to take place in-person. It might be to get you to the yoga shop or the local gym.

Food And Nutrition Importance

Food and nutrition seem to be a factor in extending life.

Once again, the mobility aspects of the AI self-driving car can assist.

We already have lots of ridesharing like services emerging today that will bring food to your home. The emergence of AI self-driving cars is going to certainly expand that capability. The so-called “boxes on wheels” will be food delivery vehicles that are being operated as AI self-driving cars. The ease of getting food delivered to your home will be simplified.

This all seems pretty good and an encouragement that AI self-driving cars might have another significant benefit to society, namely extending our life spans.

It is perhaps an indirect mechanism rather than a direct mechanism. I say this because the AI self-driving car itself is not per se extending life. It is the consequences of what the AI self-driving car can provide as capabilities that ties into the factors that presumably lead to longer lives.

I mention this because sometimes someone will argue that it is “unfair” to suggest that the AI self-driving car is extending our life spans – but it isn’t a pill that you swallow, it isn’t something you wear on your back like a special kind of cloak.

Yes, I agree, it’s what the AI self-driving car can otherwise do that counts here.

Other Side Of The Coin

As with anything that can be a benefit, the odds are that there will be potential unintended adverse consequences too.

The AI self-driving car could actually become a life limiter, rather than a life extender.

Suppose that the advent of AI self-driving cars allows people to take greater risks by having the AI self-driving car drive them to cliff diving or to parachute jumping.

You could use the mobility for purposes that put you at greater risk.

Maybe you have the self-driving car bring you fatty foods every day to home and to work. Perhaps you use the self-driving car to avoid having to contend with visitors by never being at home?

You might even become addicted to your AI self-driving car, which is unlikely to aid in your potential quest for longevity.

See my article about being addicted to AI self-driving cars: https://aitrends.com/selfdrivingcars/addicted-to-ai-self-driving-cars/

See my framework about AI self-driving cars: https://aitrends.com/selfdrivingcars/framework-ai-self-driving-driverless-cars-big-picture/

Conclusion

You’ve likely seen the famous sigmoid graph that shows the typical mortality rate for humans.

It’s a kind of “S” curve that starts up, then stays at a relatively constant rate of increase, and then tails off at the end.

Benjamin Gompertz was the famous mathematician that is most known as the formulator of the “law of mortality” and for which he asserted that the human rate of death is related to age as a sigmoid function. A variant is the Gompertz-Makeham law that includes the sum of age-independent components.

Is there perhaps no true ceiling for human aging?

Is the sky the limit?

Or, do we all have a stamped on us a perishable by date X that we don’t even know is there?

Gompertz’s indication that resistance to death decreases as the years increase might either be an immutable law of nature, or maybe it is something that we can defy or at least extend.

If you are looking for more reasons to want to have AI self-driving cars, one could be that it might aid our societal efforts to maximize our life spans.

Some might see this as a bit of a stretch and be upset that the AI self-driving car itself is not really doing this, and instead it is the consequence of what the AI self-driving car can possibly provide. Either way, it’s certainly an intriguing notion and one that might help us all as we struggle to get AI self-driving cars into suitable shape for aiding society.

The pain along the way might be worth the advantages it can provide once we get there.

I’ll see you on the other side of 150 years of age.

Copyright 2019 Dr. Lance Eliot

This content is originally posted on AI Trends.

Drone tech to help in rescue & relief ops

A team of students from IIT Madras has developed a drone technology that can help in rescue and relief operations, especially after natural disasters.

Will Business Intelligence Keep Us Safe on the Road?

When you lay down in bed at night and think over your day, you may consider some of the risky or dangerous things you did. Maybe you were putting up Christmas lights on a wobbly ladder or jay-walked across a busy street. Chances are, however, you haven’t even considered the most dangerous thing you do practically every day: driving or riding in a car.

Every day 3,287 fatalities occur in car accidents, making it the single most dangerous thing that any American does on a daily basis. Of all the accidents that happen every day, upwards of 25% is a result of distracted driving. With the advent of technologies that make communication easier than ever, the number of distracted driving accidents has gone up profoundly.

This may lead you to wonder how companies that are producing these technologies are responding. Are they using distracted driving data to come up with better and safer means of using their devices? Or are they sticking to the “business as usual” plan, regardless of the cost?

Impacts of Distracted Driving

In the grand scheme of things, there are three categories of distracted driving behaviors:


Visual: Looking away from the road for reasons such as checking on children in the ...


Read More on Datafloq

Will AI Change Education as We Know It?

New technologies are usually developed for the military sector, then find their way into healthcare, finance, retail, and, finally, education. AI will most likely follow the same trace, although it is still in its infancy in most domains. 

In education, we can expect AI to play a significant role when it comes to better management of administrative tasks, grading, and personalized learning. The way students learn will definitely change, and it is about time, as the current educational system is stuck in the twentieth century while the demands of the real world are much more advanced.

Some people might fear this development, but we are already reaping the benefits of AI when we watch Netflix or use routing apps like Waze. We don’t have to be scared of robot teachers and dehumanization of teaching. Instead, there are a few practical ways AI can be integrated into the teaching process to enhance it.

Better Test Grading

Automatic grading of multiple-choice tests has been around for decades, and there can be little improvement from AI. However, the real revolution AI can bring is related to evaluating free-form writing and even mathematical calculations.

Imagine that a teacher needs to grade over 100 essays in an unbiased way each ...


Read More on Datafloq

Spring House Coworking plans to raise $2M for expansion

At present, Spring House has 11 coworking operational centres, totalling 1,23,500 sq ft area and around 2,050 seats, in Gurugram, Noida, Delhi and Lucknow.

Drone bubble bursts, wiping out startups and hammering VC firms

A drone venture with backing from Kleiner Perkins and Google’s GV shut down last year. On a smaller scale, a company called CyPhy Works offers another cautionary tale.

We want to give India Inc latest R&D facilities: Ashutosh Sharma

Ashutosh Sharma said since the industry is an important component of growth and innovation, we want to provide companies with cutting-edge R&D facilities.

Govt is setting up high-tech R&D facilities for India Inc to encourage big-bang projects

The Department of Science & Technology (DST) plans to set up 15 centres with high-end science and technology infrastructure.