Friday, 2 August 2019

Announcing the Delta Lake 0.3.0 Release

We are excited to announce the release of Delta Lake 0.3.0 which introduces new programmatic APIs for manipulating and managing data in Delta tables. The key features in this release are:

  • Scala/Java APIs for DML commands – You can now modify data in Delta tables using programmatic APIs for Delete (#44), Update (#43) and Merge (#42). These APIs mirror the syntax and semantics of their corresponding SQL commands and are great for many workloads, e.g., Slowly Changing Dimension (SCD) operations, merging change data for replication, and upserts from streaming queries. See the documentation for more details.
  • Scala/Java APIs for query commit history (#54) – You can now query a table’s commit history to see what operations modified the table. This enables you to audit data changes, time travel queries on specific versions, debug and recover data from accidental deletions, etc. See the documentation for more details.
  • Scala/Java APIs for vacuuming old files (#48) – Stale snapshots (as well as other uncommitted files from aborted transactions) can be garbage collected by vacuuming the table. See the documentation for more details.

Updates and Deletes

There are a number of common use cases where existing data in a data lake needs to be updated or deleted:

  • General Data Protection Regulation (GDPR) and California Consumer Privacy Act (CCPA) compliance
  • Change data capture from traditional databases
  • Sessionization to group multiple events into a single session is a common use case in many areas ranging from product analytics to targeted advertising to predictive maintenance.
  • Deduplication of records from sources

Since data lakes are fundamentally based on files, they have always been optimized for appending data than for changing existing data. Hence, building the above use case has always been challenging. Users typically read the entire table (or a subset of partitions) and then overwrite them. Therefore, every organization tries to reinvent the wheel for their requirement by hand-writing complicated queries in SQL, Spark, etc.

One of the popular demands in Delta Lake is support for updates and deletes. In 0.3.0 release, we have added Scala / Java APIs to easily merge and delete records.

Delete Example

You can remove data that matches a predicate from a Delta Lake table. For instance, to delete all events from before 2017, you can run the following:


import io.delta._

val deltaTable = DeltaTable.forPath(sparkSession, pathToEventsTable)
deltaTable.delete("date < '2017-01-01'")        // predicate using SQL formatted string

import org.apache.spark.sql.functions._
import spark.implicits._

deltaTable.delete($"date" < "2017-01-01")       // predicate using Spark SQL functions and implicits

Update Example

You can update data that matches a predicate in a Delta Lake table. For example, to fix a spelling mistake in the eventType, you can run the following:


import io.delta._
val deltaTable = DeltaTable.forPath(sparkSession, pathToEventsTable)

deltaTable.updateExpr(            // predicate and update expressions using SQL formatted string
  "eventType = 'clck'",
  Map("eventType" -> "'click'")


import org.apache.spark.sql.functions._
import spark.implicits._

deltaTable.update(                // predicate using Spark SQL functions and implicits
  $"eventType" = "clck"),
  Map("eventType" -> lit("click"));

Merge Example

You can upsert data from a Spark DataFrame into a Delta Lake table using the merge operation. This operation is similar to the SQL MERGE command but has additional support for deletes and extra conditions in updates, inserts, and deletes.

Suppose you have a Spark DataFrame that contains new data for events with eventId. Some of these events may already be present in the events table. So when you want to merge the new data into the events table, you want to update the matching rows (that is, eventId already present) and insert the new rows (that is, eventId no present). You can run the following:


import io.delta.tables._
import org.apache.spark.sql.functions._

val updatesDF = ...  // define the updates DataFrame[date, eventId, data]

DeltaTable.forPath(spark, pathToEventsTable)
  .as("events")
  .merge(
    updatesDF.as("updates"),
    "events.eventId = updates.eventId")
  .whenMatched
  .updateExpr(
    Map("data" -> "updates.data"))
  .whenNotMatched
  .insertExpr(
    Map(
      "date" -> "updates.date",
      "eventId" -> "updates.eventId",
      "data" -> "updates.data"))
  .execute()

See Programmatic API Docs to understand more details on the APIs.

Query Commit History

Delta Lake automatically versions the data that you store. Delta Lake maintains a commit history about all the operations that modified the table. With this release, you can now access the commit history of the table and understand what operations modified the table.

Auditing data changes is critical from both in terms of data compliance as well as simple debugging to understand how data has changed over time. Organizations moving from traditional data systems to big data technologies and the cloud struggle in such scenarios. The new API would allow users to maintain a track of all the changes to a table.

You can retrieve information on the operations, user, timestamp, and so on for each write to a Delta Lake table by running the history command. The operations are returned in reverse chronological order. By default, table history is retained for 30 days.


import io.delta.tables._

val deltaTable = DeltaTable.forPath(spark, pathToTable)

val fullHistoryDF = deltaTable.history()    // get the full history of the table.

val lastOperationDF = deltaTable.history(1) // get the last operation.

The returned DataFrame will have the following structure.


+-------+-------------------+------+--------+---------+--------------------+----+--------+---------+-----------+--------------+-------------+
|version|          timestamp|userId|userName|operation| operationParameters| job|notebook|clusterId|readVersion|isolationLevel|isBlindAppend|
+-------+-------------------+------+--------+---------+--------------------+----+--------+---------+-----------+--------------+-------------+
|      5|2019-07-29 14:07:47|  null|    null|   DELETE|[predicate -> ["(...|null|    null|     null|          4|          null|        false|
|      4|2019-07-29 14:07:41|  null|    null|   UPDATE|[predicate -> (id...|null|    null|     null|          3|          null|        false|
|      3|2019-07-29 14:07:29|  null|    null|   DELETE|[predicate -> ["(...|null|    null|     null|          2|          null|        false|
|      2|2019-07-29 14:06:56|  null|    null|   UPDATE|[predicate -> (id...|null|    null|     null|          1|          null|        false|
|      1|2019-07-29 14:04:31|  null|    null|   DELETE|[predicate -> ["(...|null|    null|     null|          0|          null|        false|
|      0|2019-07-29 14:01:40|  null|    null|    WRITE|[mode -> ErrorIfE...|null|    null|     null|       null|          null|         true|
+-------+-------------------+------+--------+---------+--------------------+----+--------+---------+-----------+--------------+-------------+

Vacuum

Delta Lake uses MVCC to enable snapshot isolation and time travel. However, keeping all versions of a table forever can be prohibitively expensive. You can remove files that are older than the retention threshold by running vacuum on the table. The default retention threshold for the files is 7 days. The ability to time travel back to a version older than the retention period is lost after running vacuum. Running the vacuum command on the table recursively vacuums the directories associated with the Delta Lake table.


import io.delta.tables._

val deltaTable = DeltaTable.forPath(spark, pathToTable)

deltaTable.vacuum()        // vacuum files not required by versions older than the default retention period

deltaTable.vacuum(100)     // vacuum files not required by versions more than 100 hours old

Cloud Storage Support

In case you missed it, in our earlier 0.2.0 release, we added support for cloud stores like S3 and Azure blob store. The release also includes support for improved concurrency. If you are running big data workloads in the cloud, check out our previous release.

What’s Next

We are already gearing up for our next release in September. The major features we are currently working on is the python and SQL APIs for Delta DMLs and data expectations which will allow you to set validations on your tables. You can track all the upcoming releases and planned features in github milestones.

Coming up, we’re also excited to have Spark AI Summit Europe from October 15th to 17th. At the summit, we’ll have a training session dedicated to Delta Lake. Early bird registration ends on August 16th 2019.

--

Try Databricks for free. Get started today.

The post Announcing the Delta Lake 0.3.0 Release appeared first on Databricks.

More Trust from Intel Agencies Needed for AI to Work for Government

Federal Government Aims to Get is Data in Order for AI

The Top 11 Blockchains for Enterprise Organisations, and Why

Organisations wanting to use blockchain technology to decentralise their data and collaborate with industry partners are now presented with a bewildering range of options. Since the Bitcoin revolution began in 2008, the number of blockchain networks has mushroomed to the point where companies must carefully select which one is right for them.

They should start by understanding the meaning of decentralisation and the reasons why it could benefit them. The benefits of decentralisation include efficiency gains, the provenance of products and data, increase trust amongst industry partners, guarding against data breaches and redundancy against server outages. However, each company should understand why they want to decentralise their processes and data. After all, you do not require blockchain for every problem. Once that is known, you can decide upon the type of blockchain to use.

I have selected the top 11 blockchains that enterprise organisations could consider when doing this, with descriptions of why they are suitable and how they differ.

Bitcoin

Bitcoin is the original blockchain, developed as a fundamental part of Satoshi Nakamoto’s peer-to-peer electronic cash system that removed the need for a central authority to verify transactions.

With nearly 10,000 nodes worldwide working to verify transactions and keeping the ledger up-to-date, the ...


Read More on Datafloq

Can AI Solve the Biggest Obstacles to P2P Marketplaces?

Artificial intelligence is shaping the future of online commerce. One of the most underpublicized trends is the growing role of AI in peer to peer marketplaces. AI is beginning to solve a number of the most pressing challenges in the shared economy industry.

The Evolving Role of AI in Peer to Peer Marketplaces

The online shopping experience has changed remarkably over the past decade. One of the biggest changes has been the sudden popularity of peer to peer marketplaces. A recent study by PWC shows that the market for peer to peer platforms will rise to $355 billion by 2025.

 As the growth of peer to peer marketplaces accelerates, some of the shortcomings are becoming more widely understood. A growing number of peer to peer platforms are turning to artificial intelligence to solve these challenges. Some of the problems that AI can solve include:


Helping inexperienced sellers set the right prices for their listings
Detecting and eliminating spam
Removing sellers that use misleading information in their listings
Matching customers with the most appropriate listings


The potential applications of AI are virtually endless. AI will become more important to peer-to-peer platforms as it is more impregnated into their business models. Entrepreneurs aspiring to build a peer to peer marketplace ...


Read More on Datafloq

Biz in China not easy: IT cos to Piyush Goyal

The companies flagged the issue of high taxes in China, where various levies on provident fund, medical, pension and unemployment add up to a staggering 44%.

Let artificial intelligence do the health check

Tech companies are starting to see potential in cancer diagnosis and prediction, even as a clutch of homegrown startups has already come up with innovative solutions in this space.

Govt calls on industry to make robust investments in 5G tech

“There is need of robust investment of industry in 5G innovation, startups and creation of 5G products that can create India specific patents,” minister of state for communications Sanjay Dhotre said.

Network performance regressions from TCP SACK vulnerability fixes

On June 17, three vulnerabilities in Linux’s networking stack were published. The most severe one could allow remote attackers to impact the system’s availability. We believe in offering the most secure image available to our customers, so we quickly applied a kernel patch to address the issues.

Since the kernel patch was applied, we have observed certain workloads experiencing unexpected, nondeterministic performance regressions on the Amazon Web Services (AWS) platform, manifesting in the form of lengthy or hung writes to S3. Even though these regressions can be observed in less than 0.2% of cases, we wanted to share with you what we have found so far and a mitigation strategy.

Symptom

The behavior can manifest in any Databricks Runtime / Apache Spark version. Affected customers will see Spark jobs running on Databricks clusters slow down and potentially “hang” for 15 minutes, or fail entirely due to timeout, when writing to Amazon S3. Customers may see a stack trace similar to this in Databricks cluster logs:

org.apache.spark.SparkException:
... truncated ...
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception:
Your socket connection to the server was not read from or written
to within the timeout period. Idle connections will be closed.

Customers can also look at the Spark web UI and see one or more tasks taking an abnormally long time compared with most other tasks of the same stage.

Since the security patch was applied to the Databricks platform on June 24th, any change in performance would coincide precisely with that date. Jobs that performed normally on (or after) June 24th and later experienced a performance regression would be unrelated to this issue.

Root Cause

The TCP SACK DoS vulnerability was disclosed on June 17, 2019. It enables a remote attacker to trigger a kernel panic on a server that is accepting traffic on a port.

Our Infrastructure Security Team immediately triaged the issue and decided to ship the patch to this CVE in our regular security release train. We shipped this update in the form of a new Amazon Machine Image (AMI), which forms the base image for the operating system on which we run LXC containers containing the Databricks Runtime.

Shortly after rolling out the patch, we identified nondeterministic abnormalities in a very small subset of our internal benchmarks and in some customer jobs. For example, short 5-minute data processing jobs that write to Amazon S3 were taking up to an hour to finish.

Using the reproduction from our internal benchmarks, we analyzed network traffic between Databricks clusters and Amazon S3. Clusters without the patch exhibited the expected behavior when writing to Amazon S3, consistently completing writes in less than 90 seconds:

However, the same code running on clusters with the security patch experienced short periods of data transfer to Amazon S3 followed by long periods of inactivity. Here is an example job that took over 15 minutes to complete:

Mitigation Strategy

We are still actively investigating the issue in order to determine the root cause. Fixing this non-deterministic performance regression might require another OS-level patch. In the meantime, security is our default and we will continue to ship the most secure kernel we can offer. We will share with you updates as soon as we have them.

Fortunately, Spark and Databricks’ platform have been designed from the beginning to mitigate these types of long-tail distributed system problems. Customers can turn on task speculation in Apache Spark by setting “spark.speculation” to “true” in their cluster configuration to mitigate this issue. This capability was designed initially to mitigate stragglers, in the case of machine slowdowns. When speculation is turned on, Spark will launch a replica of the long-running slow task and retry it, with a high likelihood that the replica task will finish quickly without hitting the performance regression.

For customers that do not want to leverage task speculation and can accept a different security threat model, our support team can work with you to provide alternate mitigation strategies. Please contact your Account Manager or help@databricks.com if you are affected and require assistance identifying a workaround.

 

--

Try Databricks for free. Get started today.

The post Network performance regressions from TCP SACK vulnerability fixes appeared first on Databricks.

Executive Interview: Sol Rashidi, Executive VP, Chief Data Officer, Sony Music Entertainment

Advice: Figure out What Is and Isn’t AI, Be Patient, Institutionalize What You Have Built to Gain Competitive Advantage

Sol Rashidi, a thought leader in the data, robotics, AI and IT space, She doesn’t love the term “artificial intelligence”, preferring instead augmented or automated intelligence. She’s quick to point out the data scientists, engineers and human creativity behind the “artificial” solutions. But she has also been working with AI long enough to see a future past the hype. There’s no silver bullet she warns, but the advantages are real and the number of companies solving real problems is growing. 

Rashidi is currently executive VP and chief data officer, Sony Music Entertainment. She has been issued seven patents related to data requirements, data governance and IT management. Her past positions have included chief data and cognitive officer for Royal Caribbean, partner of data, analytics and AI at Ernst & Young, and member of the IBM team that first brought Watson to market. She has a bachelor’s degree in chemistry from the University of California, Berkeley and an MBA in Strategy and Leadership from Pepperdine University. She played on the water polo and rugby teams at Berkeley, and on the Women’s National Rugby Team for several years.

Rashidi spoke with AI Trends editor John Desmond about how company leadership is reacting to AI, the role of intellectual property in the space, ethical pitfalls, and GDPR. Their conversation has been edited for length and clarity. 

AI Trends: For companies interested in pursuing AI to gain some advantage in their business or to keep up with their competition, what are the trends you see in data science and data analytics?

Sol Rashidi: I try putting my finger on the pulse of what I think the trends are, and almost in every company or organization I’ve worked with, where I’ve either driven the AI agenda or the cognitive services agenda, the answers have been completely different. I wish there was a silver bullet; I wish there were two or three things. I think what it comes down to is everyone’s looking for that competitive advantage to make sure that they survive in this ever-changing world of ours. And they don’t necessarily know what the answers are, but they’re willing to explore, they’re willing to invest.

But I think what distinguishes one company from another is the internal culture, and whether it’s set up for and supports innovation, There is a difference between driving innovative agendas, and truly being able to absorb things that need to change: identifying the culture, mindset, organizational readiness elements that need to change, ensuring that whatever AI aspects are introduced, they truly get operationalized and institutionalized. That’s something all companies are having an issue with right now as it relates to AI. While there’s a need for AI—100%—the answer is different for each company. It’s still tenuous as to whether or not companies are ready to absorb this. A lot of the stuff is forward-thinking for a lot of industries.

Can you describe some of the range of maturity you see among companies with regard to AI and data governance?

The fintech industry is doing a really good job because they’ve always been forward-thinking. They may have an innovation center, a design lab and they also have their internal structure, so that when something comes along from a lab, they do a better job of institutionalizing it. 

Pharmaceutical companies do a really good job as well. Other industries doing well are travel, hospitality, consumer products, and retail. It’s spilling over to media and entertainment as well. Certain industries are definitely leading the pack in how quickly they can operationalize the capabilities that they bring along.

Are business executives getting a handle on how to exploit AI to help their businesses?

They are getting better. The challenge that executives have—myself included—is there’s so much to research and study and understand, and there are so many sources of information that you just can’t quite put your finger on what AI really means, and how it’s changing the world. We view it as artificial intelligence, but time and time again I have said: “There’s nothing artificial about this; it’s still fingers to keyboard.” We’re still training, we’re still modeling, we’re still fine-tuning. And it’s just our brains, it’s our creativity, it’s our engineers, it’s our developers, it’s our data scientists that are really running these machines that we’re referring to when we say, “Oh, it’s an artificial solution.” I think at best there’s assisted intelligence. There’s augmented intelligence, automated intelligence. I just don’t think there’s anything artificial about it.

And, unfortunately, because all the sources are providing executives with this information, they’re almost creating this aura that if you do AI, your problems will be solved. That’s the furthest thing from the truth. It’s not a silver bullet. I think there is a disadvantage in that, if for example you’re an executive in marketing, sales, or research and technology. Your plate is so full trying to manage and maintain and keep the strategy and the vision of the company in place. But to really dive in deep and understand what AI means, you just don’t have that balance and capacity. So, by default, you have to depend on your sources.

But our sources are not doing anyone any justice because they’re painting this rosy picture and, unfortunately, it’s just creating a little bit of an unrealistic expectation of what AI does. For those of us who are actually in the space who have delivered and deployed [solutions], sometimes we have to do some level-setting and say, “Well, technically these concepts are true. In reality, this is what happens.”

How is the role of a data scientist different from what a software analyst does?

We often refer to analysts as individuals who are mining through the data to understand patterns of what’s taking shape and then producing operational reports. Here were our sales, here was our revenue, here’s our forecast of what we think based on historical data. And that’s what analysts have really done today. Back in the day, you should know, at a minimum, SQL. If you are a statistician, you would know R; it depends on your language. But now we’ve built a lot of great tools that shortcut that process. You don’t need to know SQL. It’s a lot of drag and drop functionality. From my perspective, an analyst’s job is really to report out on what has happened. 

A data scientist, however, is a completely different animal in nature. They know scripting and coding and data engineering, and their job is to predict something before it’s even happened. Their job is to put completely unrelated pieces of information together to find out if there’s a way that these things relate. Their job, essentially, is to be able to find a way to react to something that’s never been reacted to before. In my viewpoint, data scientists deal with the unknown, analysts deal with the known.

I think that is a hard concept for people to grasp, especially within our industry. The data scientist is the new buzzword title of the year, and analysts who are not data scientists are saying, “I’m a data scientist.” Organizations are naming their head of data science because they feel like if they don’t, they’re not ahead of the curve. But people really have to know the difference between the two because it’s discrediting those that really put in the time and effort to understand the language, the coding, the engineering that’s needed to be able to do the stuff. 

How important is the intellectual property being developed around AI? Are businesses willing to share their experiences around AI and business strategy? 

Not yet. Not because they don’t want to share, it’s that they haven’t figured it out just yet themselves. You have your main players in the AI space, who can have anywhere from 2,000 to 3,000 engineers whose sole job is to find IP. But for those not with the Amazons and Googles of the world, the entire open-source network that includes TensorFlow or SageMaker or just whatever tool kit is available out there, has democratized AI. But how corporations and organizations are harnessing what’s available, how they’re operationalizing and institutionalizing that stuff is TBD. It’s still a work in progress.

If they have figured it out, it makes no sense to share it because they lose competitive advantage. And if they haven’t figured it out, they still don’t want to share that because they don’t want to let anyone know that they are still figuring it out. So I don’t think companies are there yet. They have very good reasons to stay hush-hush about what they’ve done or what’s in flight right now. 

We have seen a few groups that have been quite boisterous in the marketplace, saying, “Look at all the amazing things we’re doing!” But when you look under the hood, you might not categorize that as AI or really innovation because a dozen other companies have done that before. They may not have marketed it. I think it depends on your PR strategy, quite frankly.

So the companies with an aggressive PR strategy might be willing to get the story out more?

Absolutely. Because you want to attract top talent. You want to create that demeanor, that brand in the marketplace, the leading-edge. It benefits the investors. It benefits the employees. There is a ton of benefit to it. But to what degree you are actually doing it, is a different story. Sometimes in the marketplace, perception does become reality. So one strategy a company can have is to be absolutely bullish in the marketplace about innovative things they’re doing.

On the topic of data privacy, how does GDPR—the General Data Protection Regulation of the European Union—and new privacy laws and legislation impact companies’ ability to incorporate AI into the business strategy? Is it difficult to get the required data?

Very much so. Ten years ago it was a non-issue. We thought we had volume then to do any of the deep learning techniques, but that wasn’t volume. Today’s information and the amount of data ingestion that we can do—that is volume. However, because of GDPR, the type of data we can assess, collect, analyze, has become highly scrutinized. 

In this day and age, where a ton of data is available to us, sometimes it’s not actually the data that matters. I always go back to consumers. And, by the way, I wish GDPR wasn’t a part of my job or something I have to deal with. My friendly name for GDPR the acronym is Gosh, Darn Pain in my Rear, because I have to know the legislation and the laws, to understand what we can and cannot do.

Ten years ago, we were dealing with cookies and website recommendation engines. Five years ago, it was a lot easier to collect consumer data. Our only issue was to be able to understand the behaviors; that’s what everyone was after. To understand consumer behavior better, gives business better targets, upsell and cross-sell opportunities. But now with all the privacy terms and conditions, they have to opt-in. And in Germany and Austria, the consumers have to double opt-in. If they don’t do it, you can’t use their information.

Today in order to analyze data, you need to answer: One, can we actually collect the data? Two, how do we have to store it to be compliant? Three, what words and verbiage do we have to use to ensure that when consent is given, it’s the right consent so that we can use it? Then you can get to the analysis part. And there is a fifth step. You can only use the analysis for certain means and measures. You can’t just do it for everything. So what sounds simple in nature is not anymore. It’s really, really difficult.

Are you worried at all about AI getting out of control in any area such as maybe ethics?

Yes. Everyone has a different moral compass and not everyone is built the same. Not all companies are built the same. Not all countries are built the same. And just like in human nature, there are good people and there are bad people. There are good-natured people and not so good-natured people; that exists everywhere. Because the technology has become democratized and has become available, there will be ethical dilemmas. I always say, just because something’s cool doesn’t mean we should do it. You’ve got to have the balance of cool, but not creepy. As soon as something becomes creepy and you feel it—like it’s a gut reaction—don’t do it. It’s not right. Everyone is totally different on where they draw that line in the sand of what’s creepy versus what’s cool.

Here is an example. When chatbots first came out, people were essentially asking questions and getting a response. For companies, chatbots saved limited time and resources to answer basic questions. However, not everyone uses chatbot for that reason. Companies have the option of listening in to a chat, whether it’s one Facebook messenger chat to another Facebook messenger chat, or one Skype chat to another Skype chat, or whether it’s interacting with a chatbot on a website. Those are words. In those words are letters, and all that is tracked. So, cool would be, “Let’s offer a platform to give people the means to communicate in a fast, easy, and a cost-effective means?” Creepy is, “Let’s listen in on a conversation and find out who’s talking about what, so that we can do something about it.” Depending on where your moral compass lies, I do think you can misuse the stuff that is available out there.

Any advice to readers for overcoming obstacles in working with AI?

First, be patient and don’t expect that everything you build is going to be a success. Everyone has a different maturity level or understanding of what AI really is. Patience is key. You can’t expect everyone to get it. Second, you think you can build the most amazing thing and you probably have, but it doesn’t mean it’s going to go anywhere. Expect that half your projects, in my opinion, will not become institutionalized. If you can get 50% out there, you’re doing a really good job. And third, sometimes you have to level-set the expectation. Everyone has a completely different answer and a totally different definition of what AI is, what machine learning is. We need a lot of education about AI as well as application. 

A good mentor of mine said, “You know, in this space you need a backbone and not a wishbone.” I thought that was really funny.

Good one. How well do you think the software and services industry is supporting AI right now? Are you getting what you need from the vendor community?

I see lots of pluses and minuses. The minuses are creating unrealistic expectations and building the hype around artificial intelligence. A line of code around predictive modeling doesn’t mean your solution is AI, at least not in my book. I think it takes many, many more components than that. And I think the industry has been diluted as a result. Now everyone has an AI solution and I would really challenge that. I do think vendors, in order to capitalize on the hype that’s occurred today, will add a piece to the code and rebrand something as AI. I would really question whether it really is.

On the other hand, the services and certain software companies have done a really good job of investing in the area and saying, “Listen, we [understand it as well] as we can get it right now.” As close as the market and our understanding will allow us, we’re there. They may help us be forward-leaning on some concepts we want to explore; they may have learned some lessons. I do think there are some phenomenal software companies and services companies who’ve done it more times than others and they’re really helping companies propel forward.

What do you see the future holding in this area of AI, data science and business?

There is a ton of room to grow and a lot to be learned. We have scratched the surface. All the hype will die down. Everything goes through its cycle. This is going to have a two- or three-year cycle just like everything else, then we’ll be on to the next thing. Companies will be figuring out what is and isn’t AI and how to institutionalize what they’ve built to actually give them a competitive advantage. We’re not done yet. This is not a mature space whatsoever.

Regarding the AI workforce, are you able to find the people you need in the market? Do you have any advice for young people who are interested in getting into AI? What should they study and what kind of work should they try to do?

It’s very difficult to find the AI talent, no doubt about it. The geeks are the new rock stars of today’s age. Applied mathematics, computer science, and systems engineering are having their moment to shine. Every college whether it’s community college, university, Ivy League or not, has courses and majors specifically focused on data analytics and data science. If that’s a space you want to get into, that’s a pure foundation. Pure, pure, foundation. When you want to be a business executive, you need to take economics. You need to know how numbers work. This is the same way. If you want to get into the AI and cognitive space, you need to understand how the models work, where the tool kits are, what’s used for what. How do you detect something that’s an anomaly versus something that’s a norm? If you compile all that together with some aptitude and muscle, you’re good to go.

Learn more at Sol Rashidi.

Pet Mode for AI Autonomous Cars

By Lance Eliot, the AI Trends Insider

What is the most popular type of pet in the United States? Kind of a trick question, I suppose, since you undoubtedly first thought of dogs or cats. The answer is that freshwater fish are the most popular pet, consisting of an estimated 142 million of them.

I’ll give you another try: Are there more dogs or cats as pets in the United States? 

Turns out there are about 88 million pet cats and 75 million pet dogs in the US, so cats come out to be the victor in terms of popularity by count. Dog owners would likely argue though that in spite of there being more cats, maybe we should count popularity by some other factor and dogs might therefore be considered the top dog, so to speak.

Don’t Miss It: Zombie-Car Taxes Are Arising For AI Autonomous Cars

Putting aside the debate about which kind of pet is the most popular, it is somewhat surprising to realize how big the pet market is. 

An estimated 68% of United States households have a pet. The spending on pets in the United States is an estimated $70 billion or more. That’s a lot of money. That’s a lot of households. That’s a lot of people that either own a pet or maybe enjoy being with someone else’s pet. If tomorrow somehow all pets suddenly disappeared, think about how it would impact people’s lives and how much we’ve come to rely upon having at the ready our pets.

I used to have a dog. I used to have a cat. I mention both of them since some of you might think I’m more of a dog-person or a cat-person – I’m an equal opportunity pet owner (I’ve also had freshwater fish, birds, reptiles, etc.). For my dog, I would occasionally take him down to the beach for a romp on the sand. He loved to run around and chase the birds and chase the waves as they crashed on the beach. It was quite a workout for him and he’d be tuckered out by the time we got back home.

 Getting him to the beach was a bit of a hassle. 

Animals Inside A Car Can Be A Daunting Matter

At the house he roamed free. When I took him in the car, I’d put on his leash (he usually assumed he was going for a walk around the block), and then I’d put him into the backseat of the car. Actually, the moment I opened the door of the car, he knew what was going to happen next and with grand delight he’d leap into the car. I’d use the leash to loosely tie him down so that he couldn’t wander throughout the car. As with most dogs, he enjoyed putting his nose outside the window to smell the cacophony of odors as we drove to the beach and so I always cracked open the window for him.

 Excitability would sometimes get the better of him while in the car. 

This meant that there might be biological emissions during his time in the car, including excrement and urine, which obviously is not what one usually hopes to have in their car. Anticipating these moments, I put towels in the car and a blanket upon which I hoped he would generally stay, plus I tried to train him to “hold it” until we got out of the car at our destination. The return trip from the beach was similar to the trip to the beach, except that he usually was tired and would lay down in the car. This had its own disadvantages because he often had sand on him and other muck that he might have encountered during his beach romp.

The joy of playing with him at the beach, and seeing his joy of being at the beach, made the whole gauntlet of steps to do the drive worthwhile. For my cat, trips in the car were almost always solely to take the cat to the vet. Unfortunately, the cat figured this out. As a result, the moment that I started to make motions that I was going to take the cat to the vet, the cat would hide or play hard to get. I tried a few times to let the cat be loose in the car, having a leash similar to what I had done with the dog. The cat though hated being in the car and would scratch and hiss, so the best means of transport ended up involving the use of a pet carrier for the cat (it wasn’t so much the car that the cat hated, as it was the realization that it was time to see the vet).

Whenever you have an animal inside a car, it can be a dicey proposition.

As the driver of the car, you certainly don’t want the animal to interfere with your driving. 

A cat that’s allowed to wander anywhere within the confines of the car could suddenly jump in your lap and you’d be so startled that you’d maybe steer the car off a cliff. A dog that can move around could get angry at a dog on the street and start barking, maybe distracting you, doing so just as you are making a right turn and perhaps you inadvertently hit a nearby pedestrian.

If you are transporting freshwater fish, I suppose it’s less likely of something going amiss in the car, though if you have them in a simple fish bowl, and if you happen to hit the brakes while driving, they might go flying throughout your car, and in so doing disturb you that you become distracted from the driving task. 

Anything can happen.

A rule-of-thumb would seem to be that for any animals inside a car, it’s best to control them in a manner that they cannot disturb the driving of the car.

I had a friend that would carry his dog in his arms and ride as a front-seat passenger in someone else’s car. One day, the dog freaked out when another car honked its horn, which caused my friend to reflexively try to re-grab his dog, which caused my friend to flail around in the passenger seat, which then he unintentionally hit the driver, and the driver of the car then rammed into a car ahead of them. Quite a story to tell the police or the insurance company.

The story is helpful because it highlights that there is a range of “control” that one might have with the animal. Some people tie down their pet while it’s in the car and try to immobilize it from impacting the driver. Some think that verbal commands alone will keep the animal from getting out of hand. My friend thought he could just hold his pet in his arms. My having put the cat into a pet carrier pretty much prevented the cat from disturbing my driving, though the cat meowing and hissing did admittedly perhaps distract me somewhat from being fully attentive to the driving task.

 Impacts Of Having An Animal Inside A Car

Here’s some possibilities that we want to presumably avoid when transporting an animal inside a car:

  •             Animal endangers the driver
  •             Animal endangers other occupants
  •             Animal endangers the car
  •             Animal endangers itself
  •             Animal endangers others outside the car

 I saw a dog leap out a window of a car one day and chase after a person that the dog apparently disliked, thus, it’s conceivable that an animal could endangers others outside of the car. At supermarkets, you sometimes see people that have parked their cars and left an angry dog in it, which when people walk past the car, the dog tries to take a bite out of them. Not good for anyone, humans nor pet.

 There have been cases of animals inside a car that wreaked havoc on the interior of the car. 

I even heard one time that a person left their engine running, got out of the car to do some quick task, and the dog somehow bumped into the transmission knob and the car went into drive.

Luckily, miraculously, I don’t believe anyone was hurt in that instance. I did hear though that the dog went to the Department of Motor Vehicles (DMV) in hopes of getting a driver’s license. 

Well, maybe not.

I know that some people get upset when I say that you should control your pet while it’s in your car. 

They argue with me that if the animal is domesticated, there should not be any concern about controlling the animal. Only if the animal is a wild animal do they think that there’s any need to be overtly thinking about controlling the animal. I don’t want pet owners to get upset with me, but I’ll just point out that even the most domesticated animal still has animal instincts and reactions.

In my view, you can’t be too careful, especially when it comes to putting an animal into the rather confined space of a car and once the car gets into motion you are opening up a can of worms. It can be an explosive and dangerous combination.

Pets can do any of these things:

  •         Become scared
  •         Get startled
  •         Become upset
  •         Get angry
  •         Try to run or scamper
  •         Bite
  •         Become confused
  •         Scratch
  •         Etc.

 If anything, I’m often concerned not just for the human driver or the human occupants, but also for the animal itself. When someone thinks they are doing a good thing by letting their dog roam freely in a moving car, they aren’t thinking about the harm that can come to the dog. Suppose the driver suddenly hits the brakes? That dog is going to go flying in the car and possibly get injured or killed by hitting something in the car or maybe even getting thrown outside of the car. Most states have various laws and regulations about restraining your pet while it is in your moving car, for its safety and your safety.

 Autonomous Cars And Pets Going For A Ride

What does this have to do with AI self-driving cars?

At the Cybernetic AI Self-Driving Car Institute, we are developing AI software for self-driving driverless autonomous cars. 

This includes encompassing various “edge” problems such as the transporting of pets (animals) while in an AI self-driving car.

Let’s consider some of the aspects involved in this edge problem. I call it an edge problem because it is considered by most of the automakers and tech firms as something outside the core of what an AI self-driving car is supposed to do. They are focused on getting the AI to drive the car. The aspects of dealing with any pets inside a car is considered secondary and much lower on the list of crucial things to get done.

 For my article about edge problems in AI self-driving cars, see: https://aitrends.com/selfdrivingcars/edge-problems-core-true-self-driving-cars-achieving-last-mile/

 For my framework about AI self-driving cars, see: https://aitrends.com/selfdrivingcars/framework-ai-self-driving-driverless-cars-big-picture/

 Let’s also define what is meant by an AI self-driving car. There are various levels of self-driving cars. At the topmost level, Level 5, it’s a self-driving car for which the AI can fully drive the car. This means that there is no human driver required in the self-driving car at a Level 5. For the levels less than a Level 5, there is a need to have a human driver present. The human driver and the AI are considered co-sharing the driving task, though the human is also considered ultimately responsible for the actions of the car. This notion of co-sharing the driving task is something I’ve mentioned many times is raft with various drawbacks.

 For my article about the co-sharing of the driving task, see: https://aitrends.com/selfdrivingcars/human-back-up-drivers-for-ai-self-driving-cars/

 For the levels of self-driving cars, see my article: https://aitrends.com/selfdrivingcars/richter-scale-levels-self-driving-cars/

 In the case of an AI self-driving car that is less-than a Level 5, there needs to be a human driver at the ready to drive the car. This implies that if you do have an animal in the car, it’s similar to the situation today of having an animal in a conventional car. Anything that the animal does that disturbs the human driver can have adverse and dire consequences.

 I realize you might be thinking that if the AI is co-sharing the driving task, why doesn’t it just wrench control from the human driver if the human driver suddenly becomes unable to do the driving task.

 This has several problems.

 One is that how will the AI realize that it is best to take control from the human driver? Even if the AI is detecting whether the human driver has their hands on the wheel or maybe via a camera whether the human is looking forward at the road, trying to judge when it is appropriate to take over control is not very transparent. Imagine too if the AI does take over control and it turns out that the human was still in control, but that the AI now maybe is going to take an untoward action since it doesn’t know what the human driver was intending to do.

 You could even get into a tug of war between the AI and the human, trying to take control from each other. I suppose you might contend that if the human tries to take control back, the AI should relent. But, suppose the human doesn’t know now what the AI was intending to do, and so the human puts the car into an untoward posture. Or, maybe the human takes back control, but then let’s say the dog in the car jumps on the human a second time and it is necessary for the AI to once again take control. Meanwhile, maybe you’ve used a rule that if the AI takes control, and the human takes it back, leave the control with the human, avoiding a tug of war. That wouldn’t work out in all cases.

 You might say that if the AI is indeed watching what’s going on in the self-driving car, maybe it should be sharp enough that it can figure out that say a dog is loose or cat is loose. Yes, you could potentially have the machine learning recognition that would be programmed for this, but it is much subtler than you might think. Generally, it would require some kind of common sense reasoning to try and decide whether the dogs or cats are doing something relatively safe or unsafe.

 For my article about common sense reasoning in AI self-driving cars, see: https://aitrends.com/selfdrivingcars/common-sense-reasoning-and-ai-self-driving-cars/

 Another perspective is that you might have the human driver tell the AI to take over control. This would be similar to when you initiate say Alexa and so you say a code word, “Alexa” and then you provide some kind of command. It could be that the AI self-driving car has a code word, let’s use “Lance,” and then after you say that word you can tell it to take over control of the driving task. I realize that some of you will say that suppose a child in the car suddenly yells out the code word and then all of a sudden the AI construes whatever is said next as a command (“go off a cliff”). Some counter-argue that the AI could be using a voice fingerprint identify capability such that it would only recognize and acknowledge the human driver’s voice at the time of the driving task.

 In recap, here’s some aspects involved:

  •         AI tries to ascertain if an animal is amuck
  •         AI potentially takes over driving task if human driver seems unable
  •         Human driver can signal to the AI to take over the driving task
  •         Human driver can signal to take back the driving task or refuse to give it up
  •         Miscommunication in the co-sharing could have dire consequences
  •         Misunderstanding in the co-sharing could have dire consequences

 Pet Mode For An Autonomous Car

One approach too involves having the AI be a kind of an alert monitor about the animal in the self-driving car. 

In essence, the AI could be placed into “pet mode” and be ready for the particular dynamics of having a pet inside the self-driving car. This pet mode could be initiated overtly by the human occupant telling the AI to go into pet mode, or it could be figured out by the AI via machine learning recognition of the “objects” inside of the self-driving car (and possibly with a confirmation to the human driver that indeed there is an animal on-board).

Part of the “pet mode” could be that the AI would be on the watch on behalf of the human driver about things that the pet is doing. It would be akin to having a passenger in the car that can tell you that the dog just chewed the backseat armrest, or the cat is curled up in a ball on the floor. With a human passenger, they would be able to take physical action when needed and help restrain the animal, but there’s not much the AI of the self-driving car can do in that regard. Instead, the AI would be devoted to warning the human driver about what the animal is doing, and being another pair of eyes, so to speak, while the human is presumably watching the road and being at the ready for the driving task.

You could potentially have the AI try to talk to the animal. Perhaps your pet dog has been trained to listen to the AI of your self-driving car. The AI then could potentially tell the dog to sit down or get into the backseat. I realize this seems kind of wild as an approach, but as you’ll see in a moment, maybe it’s not as crazy as it seems.

Let’s now consider the use of “pet mode” for a Level 5 self-driving car. In the case of the Level 5, the AI is doing all the driving. There is no human driver. This is handy because it implies the animal is unable to distract the driver. Whatever the animal does, the AI is still going to be able to drive the car.

The only way the animal presumably can disrupt the driving would be if it is able to damage something inside the self-driving car that could hamper the AI or the car itself. Suppose that there is wiring just under the dashboard and somehow the dog gets to it and chews through those cables. If they are involved in the electronics of the car and any kind of driving related task, it could be disruptive to the AI and the car integrity.

This then brings up facets about the interior design of the self-driving car. It is generally envisioned that since there is no longer a need for a human driver in a Level 5 self-driving car, we can redesign the interior compartment of the self-driving car. No need to have a seat facing forward in the same place that today’s driver seats reside. Instead, the compartment can be perhaps seats that swivel and face each other. Or, maybe seats that can recline fully so you can sleep in your self-driving car when you want to do so. With the potential of true AI self-driving cars being used non-stop 24×7, it is presumed that people will likely sleep while on their way to work or on trips to visit in-laws, etc.

For my article about non-stop AI self-driving car use, see: https://aitrends.com/selfdrivingcars/non-stop-ai-self-driving-cars-truths-and-consequences/

When redesigning the interior of cars to be suitable as a self-driving car, one additional consideration will be the nature of the occupants and what they might do inside the self-driving car. If you are going to put your children into a true AI self-driving car in the morning so that they will be driven to school, doing so without any adult supervision inside the self-driving car, you want to know that the children hopefully cannot harm themselves by poking around within the interior of the car. Today’s cars leave all sorts of metal joints and prods fully exposed inside the compartment, which a small child without supervision could easily harm themselves on. The same could be said about pets that are unsupervised.

Indeed, when you consider that 68% of U.S. households have pets, and once there are true AI self-driving cars prevalent, just imagine how many of these households will opt to send their pet by itself in the family AI self-driving car to go visit the vet. Or go visit grandma. Or go to a pet playground where there is someone to supervise your pet, and then they put your pet back into your AI self-driving car, which comes to work to pick you up at the end of the day, and your joyful dog is right there to greet you. No need to wait until you get home to get some hugs and kisses from your beloved pet.

The redesign of the interior of a car should take into account the notion that today’s designs are inherently dangerous and assume that whomever is in the car will be somewhat supervised. True AI self-driving cars won’t necessarily have an adult in the car to undertake supervision of children and nor pets. Overall, this implies that the interior has to be made safety proof with regard to whomever is inside the self-driving car. There shouldn’t be any easy way to cut yourself. There shouldn’t be any easy way to undermine the capabilities of the self-driving car by hitting something or chewing on something.

This does not mean that the interior needs to be a steel tank or barren fortress. The automaker can provide various covers and shields to allow for a relatively impervious interior. I’d even assert that if the automakers don’t do so, a thriving third-party market will likely develop to outfit your interior compartment so that it is safer for the transport of children, pets, etc.

I had mentioned earlier that the AI might be made to talk to the pet. Suppose your pet will be in the true AI self-driving car for an hour or two, perhaps taking a lengthy journey. That’s a long time for your pet to be alone. The AI could be talking to the pet, maybe playing music, or otherwise try to comfort the animal. Given that there are likely cameras pointing inward, you can do a Skype like chat with your pet, and the inside of the AI self-driving car is likely to have screens, usually used to show movies or do your digital work. Your pet could see you, you could see your pet, and try to comfort your pet during part of its journey in the self-driving car.

I had mentioned earlier that the seats in a true AI self-driving car might swivel and there might be other variations in terms of configuring the internal compartment. For those that have pets, there might be ways to re-configure the inside compartment to allow for taking out the seats and allowing the pet to wander around inside the car.

Perhaps there might be a special leash or restraint system that keeps the animal relatively safe, but also allows for open movement while in the car, most of the time (the restraint system might opt to more strongly restrain the animal if getting into rough traffic, or maybe if the pet is getting out-of-hand). Anyway, you can just see the ads now, the XYZ auto maker comes out with a pet friendly AI self-driving car and tries to lure buyers that have pets – could be a sizable segment of the market.

For my look at how marketing of AI self-driving cars is likely to occur, see: https://aitrends.com/selfdrivingcars/marketing-self-driving-cars-new-paradigms/

It is anticipated that the advent of true AI self-driving car is going to be a tremendous boon to the ridesharing industry. People will buy a self-driving car, realize that it can be used 24×7, and opt to rent it out while they are at work or asleep. Other people will buy a self-driving car solely as a ridesharing revenue maker and not even use it for personal purposes. We’re heading toward a ridesharing economy, or ridesharing-as-a-service world.

That being the case, what about pets? People are going to want to have their pets go lots of places that right now involves too much of a hassle for them to drive their pets to, and yet with a true AI self-driving car it could be a breeze. Some ridesharing services might tout that they have pet-devoted AI self-driving cars, ready for the transporting of your favorite dog, cat, or fish. I’ve already predicted that with the advent of AI self-driving cars we are going to see all sorts of induced demand. This is demand for using a car that otherwise today is suppressed or that people don’t even think about currently.

For my article about induced demand and AI self-driving cars, see: https://aitrends.com/selfdrivingcars/induced-demand-driven-by-ai-self-driving-cars/

Conclusion

I began this discussion by pointing out that we today spend $70 billion on our pets, doing so for pet food, pet toys, pet care, and the like. It seems logical and inevitable that true AI self-driving cars are going to ultimately intersect with our desire to have pets.

Right now, if I told you that someday your pet will be driven around in an otherwise empty car, you’d think I was loco or that it would have to be some crazy rich person that is spending way too much money on their pet. In the future, the prevalence of AI self-driving cars will open the avenue for considering ridesharing involving our pets. Easily, readily, at a low cost. It’s going to happen.

The idea of a pet mode for an AI self-driving car is not particularly far fetched if you have a long-term view, at least that’s what we say.

Well, I suppose it could be that me and my team just love our pets so much that we insist on somehow getting them involved in AI self-driving cars. 

I wonder if I could train a dog to do AI coding? 

Or, would a cat do a better job at it? 

It’s hard to say.

Copyright 2019 Dr. Lance Eliot 

This content is originally posted on AI Trends.

Ripple Effects of Driverless Cars Touch Ethics, Spending, Policy

5G Will Require AI-Enabled Cybersecurity to Handle New Threats

By AI Trends Staff

South Korea has rolled out the world’s first 5G network, with speeds 20 to 100 times faster than 4G. Along with the anticipated rollout of 5G in other countries, which will bring a boost in users of higher-capacity wireless devices, are security concerns.

Many organizations will have to change or restructure their cybersecurity strategies to deal with the new platform, suggests an account from  Malwarebytes Labs

The account identifies ways the rise of 5G can have an impact on a company’s cybersecurity. For example: 

New risks will surface. By way of comparison, the Mirai botnet in 2016 executed a denial-of-service (DDoS) attack that took down most of the internet on the east coast. The attack spread through thousands of insecure IoT devices, including security cameras. The creator of Mirai had intended to take down rival Minecraft servers, in order to make more money. The effect on the overall internet was an unintended consequence. When 5G networks roll out, devices will be powered on and connected from a variety of mediums, increasing risk.

Increased Bandwidth Creates New Threat Opportunities. Many of today’s security services monitor traffic in real time to identify threats based on activity and sniffed data. For example, if someone in-house is visiting a flagged URL, it could be an inside threat. Because network capacity, security and traffic can be managed today. With 5G speeds and capacity, that may no longer be the case. It could be that many of today’s cybersecurity solutions may no longer work in the 5G network.

Security Automation May Be Required. Security automation and integration is when the security architecture and system in use is connected across the operation. Data must sync between security layers. Attackers are likely to use physical means and digital means to attack, and move between the two. They are likely to use a combination of strategies and attacks to gain unauthorized access. This has been shown by Emotet’s polymorphic, multiple module attacks or CrySIS ransomware’s versatile attack vectors. (Users: Do not click on suspicious attachments, and use strong passwords.)

Complexity of 5G Network Security Will Make AI Required

AI will be required to secure 5G networks, suggests an account from threatpost. That was an assessment from the GSMA Mobile 360 Security for 5G conference held in May in the Netherlands. 

The authors set the stage by noting that the existing telecom networks are built from a hardware-centric perspective, using the vertical-stack Open Systems Interconnection (OSI) model. This includes a heavy reliance on hardware big routers and switches with device-specific software. Functions are hard-coded. Extensive support systems are needed to carry out management and orchestration functions. 

In contrast, 5G takes a page from the world of enterprise IT and the cloud, according to Brian Wagner, head of security, risk and compliance for EMEA at Amazon Web Services. “Security is no longer a silo that sits in a separate area,” he said, during a keynote at the conference. Security is not compliance anymore, and vice-versa. These networks will consist of largely commoditized technology. So, you have to upgrade your tools, take a risk-based approach and be transparent.”

In a 5G network, hardware servers are abstracted from the software; all functions are virtual; a packet core network is software-defined and programmable, able to make changes to services on demand. Thus, 5G networks will be capable of supporting literally billions of endpoints generating data, all with potentially custom network services.

Achieving visibility into this very different and more complex environment is something network managers have not had to do before. 

AI will be required for intelligent, adaptive security management and automation, the speakers suggested. The technical switch to a software approach and dynamic updating will be a challenge. The AI algorithms will need to be trained with a tremendous volume of security knowledge, suggested Martin Borrett, CTO and engineer at IBM Security, who has been working with the company’s AI platform, Watson.

Read the source articles in Malwarebytes Labs and  threatpost.

AI Being Enlisted to Help Monitor Energy Consumption of Individual Households and Vast Businesses

By AI Trends Staff

Energy monitoring can range from knowing about how much energy is being consumed in a household to how much is needed to power a business. The domain of energy consumption is proving ripe for exploitation by AI.

In a recent study conducted by the US Energy Information Administration and reported in VentureBeat, four percent of owners of smart home electric meters reported viewing their hourly or daily energy consumption data. That translated to business opportunity to Mike Phillips, Christopher Micali and Ryan Houlette, who in 2013 cofounded Sense, to develop a platform that taps AI for real-time insights on electrical usage. The Cambridge, Mass.-based company recently raise another $10 million in a Series B investment round, to bring its total B round investment to $30 Mill and its total raised to $50 million.

Previous investors included Schneider Electric and Landis+Gyr. “These additional investments reflect our shared vision of a smart, energy-aware home. From the company’s founding, we envisioned a future when homes would share information about themselves with their owners, who could make better decisions about how to live in today’s world.” said CEO Phillips.

Landis+Gyr is planning to integrate Sense’s technology with its Gridstream Connect IoT platform for utilities. Sense’s home energy app, which monitors energy consumed by electrical devices, will be made available as a plugin for the Connect IoT platform, enabling consumers to see how much energy appliances, lighting and other devices are consuming in real time and recommending ways to cut down on usage.

Schneider Electric has built Sense’s technology into its Wiser Energy System by Square D, and through a partnership with a solar financing provider. The collaborators have signed on more than 200 solar installer partners to date. 

Sense’s technology was developed by a team from Philips, Amazon, Nuance, Vlingo, ScanSoft and SpeechWorks. The company’s $299 home energy monitoring system features a combination of sensors that connect to breakers inside the home’s electrical panel, and a compute box that links to remote services.

Competitors to Sense include Smappee and Currant. Revenue from consumer technology and services for home energy management is anticipated to grow to $7.8 billion by 2025, according to Navigant Research.

Overcoming Persistent Barriers to Renewable Energy

Renewable energy such as solar and wind accounted for 10 percent of all energy consumption in the US in 2016, reported the US Energy Information Agency. Barriers to wider implementation persist. Researchers are exploring how artificial intelligence cold assist in improving the accessibility to renewable energy. A recent account in emerj examined three major categories of renewable energy technologies that incorporate AI, aimed to appeal to business leaders interested in green energy.

For energy forecasting, energy provider Xcel Energy of Colorado seeks to address the challenge of fluctuating solar and wind power sources. The company’s produsting using an AI-based data mining method made available by the National Center for Atmospheric Research. It uses a combination of data from local satellite reports, weather stations and wind farms, to identify patterns and make predictions. 

Xcel’s plans to expand wind generation by 50 percent by the year 2021.

For energy efficiency, Verdigris Technologies of California is offing a cloud-based platform leveraging AI to help clients optimize energy consumption. The offering is designed for large commercial buildings and managers of enterprise facilities. Smart sensors in IoT hardware are directly attached to the client’s electrical circuits to track consumption. The data captured is sent to the cloud in a secure manner, and presented to the client on a dashboard. 

Since its founding in 2011, Verdigris has raised over $16.5 million, including from Verizon Ventures in 2016. The company reported working with the W Hotel in San Francisco to identify energy inefficiencies in the hotel’s commercial kitchen. Within three months of use, the system had identified inefficiencies costing the hotel more than $13,000 annually.

For energy accessibility, PowerScout uses AI on industry data to model potential savings for homeowners on utility costs. The product identifies smart home improvement projects based on energy usage, and matches clients to potential installation providers in an online marketplace. 

PowerScout has received two grants from the US Department of Energy amounting to a total of $2.5 million.

Read the source articles in VentureBeat and emerj.

Launching an AI Project Calls for Careful Scoping

Grocers Wading into a Future with AI

By AI Trends Staff

The grocery story business is beginning to use AI to try to gain a competitive edge. Salt Lake City-based Associated Food Stores (AFS), for example, has 500 stores in the western and southwestern US. It found itself dealing with a growing number of SKUs that stores managers were having difficulty tracking and prioritizing, according to an account in ChainStoreAge.

AFS began using an AI solution from CB4 to analyze point of sale data, to identify when physical issues in a store are hold back sales. These could be products not easily visible and out of stock conditions. CB4 guides them to how to fix potential issues with the exact SKUs in their stores that need the most attention. No in-store hardware or external data sources are required. 

“What really caught my attention was the ease of implementation,” said Wade Judd, CIO of AFS. “We were able to roll out some pretty sophisticated technology, train our store managers, and start reaping the benefits in weeks rather than months. It was one of the simplest implementations that our team has been involved with.”

Grocery stores may not be on the bleeding edge of AI innovation, but new survey data shows the pace is picking up. 

An account in Winsight Grocery Business cites the Technology Vision 2017 Consumer Goods Report from Accenture, reporting 78 percent of industry executives in agreement that AI will revolutionize customer interaction.

“It’s weaving its way into nearly everything,” says Gary Hawkins, CEO of Los Angeles-based Center for Advancing Retail & Technology (CART). “Retailers have realized the world is changing really fast and they’re not competing just with the store down the street but with Amazon, Walmart, Kroger, who are investing tens of millions of dollars in technology every year.”

The account was rife with examples. Harps Foods in Springdale, Ark. has been working with Daisy Intelligence, from Concord, Ontario, to improve pricing and promotions in its circular. The company used its AI to analyze two years of transactional data and pricing from Harps; the goal is to grow sales by 3 percent.

Daisy is also working with Earth Fare, a 49-store chain headquartered in Fletcher, NC, which sought help in featuring the right products in its flier. Comparable store sales have grown five percent each year in the three years Daisy has been working with Earth Fare.

AI company Revionics enables price optimization, allowing stores to change prices as frequently as overnight.

Ahold Delhaize is a Dutch retailer with US brands including Giant, Stop & Shop, Hannaford’s and PeaPod. The company announced a partnership in the spring of 2018 with the Innovation Center for Artificial Intelligence. CEO Frans Muller was quoted as saying the company wanted to “learn how AI can be used to better serve the interests of our customers.”

One of those ways is by deploying robots from Badger Technologies of Kentucky into supermarkets. The Badger fleet is now in 500 Ahold Delhaize stores where they are known as “Marty.” The robot is being used primarily to spot hazards, such as a blueberry on the floor in the produce department. The robots are capable of scanning shelves for inventory and pricing, but high-resolution images and data need to be processed by the in-store network, a gating factor on the rollout.

That some supermarket customers view Marty — who is currently mute, six-feet tall and with googly eyes — as “creepy” may have to be factored in at some point in the executive suite of Ahold Delhaize.

Read the source articles in ChainStoreAge and Winsight Grocery Business.

Thursday, 1 August 2019

Making AI Work in the Real World

Figure_Eight_Appen

Scheduled for September 17, 2019

Register Today!

If you’ve read your fair share of tech press, you’ve certainly been exposed to breathless forecasts about the promise and power of artificial intelligence. The thing is, a lot of those articles are light on detail or focus too heavily on algorithms and not on business value.

In this webinar, Laura Horvath, Head of Product Marketing at Figure Eight, takes an industry-by-industry perspective on true AI adoption.

We will cover:

  • An approach to AI that realizes business value
  • Real-world examples of businesses using AI to improve their bottom line
  • Real-world use cases in e-commerce, enterprise software, robotics & IoT, AgTech, and more

We will separate the hype from the reality, the theoretical from the practical, and the research labs from ROI.

Learning objectives:

  • Real world uses for AI that improve the bottom line

Laura-HorvathLaura Horvath, Director of Product Marketing, Figure Eight
Bio To come



Register Today!

AI Can Play Poker, but It Can’t Play the Markets Yet

Facebook’s AI-powered Pluribus system astounded the world when it recently defeated six professional poker players - including World Poker Tour title record holder Darren Elias – in games of no-limit Texas Hold ‘Em. The AI ‘revolution’ is being lauded in pretty much every sector you can imagine from healthcare and agriculture to fast-moving consumer goods and even reading the news.

Worldwide spending on AI systems grew to nearly $35.8 bn this year – a 44% increase in 2018. It’s expected to more than double to $79.2 billion by 2022. In my own sector of international capital markets, the chorus of voices proclaiming the dominance of machine-learning is growing by the day.

After retail, the banking and finance sectors are the second biggest beneficiaries of AI investment. Such levels of spending are undoubtedly going to change the make-up of international capital markets and the traders who operate in them. But we must ask ourselves if we are headed in the right direction.

That AI is so popular with venture capitalists now is clear, but what’s not clear is whether it’s working. For every dollar invested in new technology, a dollar is quickly withdrawn from a project that fails to make money. In many cases, investment pours into ...


Read More on Datafloq

Is Virtual Reality eCommerce Dead?

Virtual reality is among the most hyped technologies of the last three decades. It’s been the next big thing since I was a child. I remember excited TV reporters donning huge VR headsets to navigate blocky and pixelated landscapes as far back as the nineties. But, in the last three or four years, it looked as if the hype was becoming a reality. Companies like Facebook and Microsoft were plowing big money into VR initiatives. Silicon Valley venture capitalists wrote huge checks for VR startups. Major movie studios created VR production units. Pundits declared that we were entering the VR age and that every industry would be transformed, including eCommerce.

But the VR hype seems to have died down. Companies like Facebook’s Oculus continue to iterate on their hardware, but there’s less palpable excitement. Investment in VR startups has declined. The VR gold rush saw billions of dollars invested in startups focused on creating VR hardware and software. In 2018, Silicon Valley investment fell by 81%. Across the US, funding for augmented and virtual reality declined by 46% to $809 million. A lot of money, but a fraction of what we’d expect if VR had taken off in the way the ...


Read More on Datafloq

Plugin Management Library and CLI Tool Phase 2 GSoC Updates

At end of the first GSoC phase, I announced the first alpha release of the CLI tool and library that will help centralize plugin management and make plugin tooling easier.

Phase 2 has mainly been focused on improving upon the initial CLI and library written in Coding Phase 1. In particular, we’ve been focusing on getting the tool ready to incorporate into the Jenkins Docker Image to replace the install-plugins.sh bash script to download plugins. This work included parsing improvements so that blank lines and comments in the plugins.txt file are filtered out, allowing update centers and the plugin download directory to be set via environment variables or CLI Options, creating Windows compatible defaults, and fixing a bug in which dependencies for specific plugin versions were not always getting resolved correctly.

In parallel to getting the tool ready for Jenkins Docker integration, Phase 2 saw the addition of several new features.

Yaml Input

In addition to specifying the plugins they want to download via the --plugins CLI option or through a .txt file, users can now use a Jenkins yaml file with a plugins root element.

Say goodbye to the days of specifying incremental plugins like incrementals;org.jenkins-ci.plugins.workflow;2.20-rc530.b4f7f7869384 - you can enter the artifactId, groupId, and version to specify an incremental plugin.

Yaml Input Example
Yaml CLI Example

Making the Download Process More Transparent

Previously, the plugin download process was not very transparent to users - it was difficult to know the final set of plugins that would be downloaded after pulling in all the dependencies. Instead of determing the set of plugins that will be downloaded at the time of download, users now have the option to see the full set of plugins and their versions that will be downloaded in advance. With the --list CLI option, users can see all currently downloaded and bundled plugins, the set of all plugins that will be downloaded, and the effective plugin set - the set of all plugins that are already downloaded or will be downloaded.

List CLI Option Example

Viewing Information About plugins

Now that you know which plugins will be downloaded, wouldn’t it be nice to know if these are the latest versions or if any of the versions you want to install have security warnings? You can do that now too.

Security Warning CLI Option Example
Security Warning CLI Option Example

Next Steps and Additional Information

The updates mentioned in this blog will be released soon so you can try them out. The focus of Phase 3 will be to continue to iterate upon and improve the library and CLI. We hope to release a first version and submit a pull request to Jenkins Docker soon. Thanks to everyone who has already tried it out and given feedback! I will also be presenting my work at DevOps World in San Francisco in a few weeks. You can use the code PREVIEW for a discounted registration ($799 instead of $1,499).

Feel free to reach out through the Plugin Installation Manager CLI Tool Gitter chat or through the Jenkins Developer Mailing list. I would love to get your questions, comments, and feedback! We have meetings Tuesdays and Thursdays at 6PM UTC.

DataRobot Raises Another $200M to Pursue Automated Machine Learning

DataRobot of Boston has raised approximately $200M in a Series E funding round to help accelerate development of its automated machine learning and AI software, according to a recent report in xconomy.  That makes the company’s total venture funding over $400 million.

The company is now in “unicorn” territory with a valuation over $1 billion. 

DataRobot was founded in 2012 by data scientists from Travelers casualty insurance. CEO Jeremy Achin had been director of research and modeling at Travelers; DataRobot co-founder Tom de Godoy, now the firm’s CTO, was senior director of research and modeling for Travelers.

The DataRobot platforms aims to automate data science functions enterprises and aggressively pursuing. In banking, for example, the software could be used to automatically assess credit default risk and track fraudulent transactions. In healthcare, it could be used to estimate hospital readmission risk.

Read the source article at xconomy.