Dhiraj Rokade

Thursday, 27 February 2020

Marketers Must Demystify AI to Reap Its Benefits

There is not a single industry that has not been touched by artificial intelligence. One of the fields that has been most affected is the marketing profession.

Most marketers appreciate the benefits that AI offers. A 2017 report by Salesforce found that 51% of marketers are using AI in some capacity. My own anecdotal experiences suggest that figure has increased to around 80% this year.

However, many marketers are still unclear about the capabilities of AI. They also aren’t entirely sure what technology should be classified as artificial intelligence.

The most competent marketing companies are seeking greater clarity from vendors that use technology based on machine learning algorithms. Companies that use AI to facilitate marketers should provide better explanations. Marketers must also familiarize themselves with the opportunities and constraints of artificial intelligence. Otherwise, they risk overpaying for solutions, getting duped by “AI snake oil salesman” or setting unrealistic expectations for their campaigns.

Knowledge gaps limit the effectiveness of AI in marketing

The Content Marketing Institute did a small meta-study of various surveys on the use of artificial intelligence in marketing. They discovered some major inconsistencies across surveys. This seems to suggest that marketers are not highly knowledgeable about the nature of this new technology.

This raises ...

Read More on Datafloq

The Rise and Rise of Hybrid Cryptocurrency Exchanges

The launch of Bitcoin in 2009 kickstarted the cryptocurrency revolution. This new digital asset was not affiliated to any central authority and allowed users to conduct transactions in an anonymous as well as secure fashion. It was not long before other cryptocurrencies emerged, each targeting a niche industry or boasting of an elaborate consensus algorithm. However, cryptocurrencies being virtual resources could not be swapped on traditional physical platforms. Thus, the requirement for specialized trading platforms was highlighted and cryptocurrency exchanges were born.Cryptocurrency exchanges conduct business the same way forex trading functions, i.e. one currency is exchanged for another currency. For example, just like how the US Dollar is traded for the Indian Rupee, a Bitcoin can be exchanged for Ripple and vice-versa. These platforms sustained themselves by levying a small fee on every transaction. Soon, these cryptocurrency exchange development were divided on the basis of the underlying architecture they followed. The two groups were defined as, Centralized Exchanges (CEX) and Decentralized Exchanges (DEX). As the name implies, Centralized Exchanges are run by third-party operators who coordinate exchanges between the buyers and sellers. On the contrary, Decentralized Exchanges stay true to the principles laid down by Satoshi Nakamoto and therefore do not ...

Read More on Datafloq

Whitelabel P2P Crypto Exchange Insights

Blockchain and cryptocurrency have always been associated with the concept of decentralization. However, when it comes to the cryptocurrency exchanges, the concept of decentralization needs to take a backseat. Peer-to-peer cryptocurrency exchanges, commonly abbreviated as P2P crypto exchange, present themselves as perfect solutions to this challenge. Let us learn about these P2P crypto exchanges and the benefits that they bring.
The Classic Crypto Exchange
Usually, when it comes to cryptocurrency exchanges, there is always an intermediary involved in executing the trade and transactions.
Users who wished to exchange cryptocurrency would post an order with details such as the quantity and unit price. These details would be added to a shared ledger called the order book. On the other side, users who would like to buy those listed coins would select an order that matched their commercial requirements. The trading process begins only after the buyer and seller requirements match.
This brings about a few disadvantages. Some of the most prominent ones are the disclosure of identity, excessive consumption of time, the possibility of fraud, and of course, the cost involved.
The most significant disadvantage brought about by the centralized exchanges lies in the very name! In essence, the core philosophy of blockchain being decentralized was ...

Read More on Datafloq

How Data Cleansing Helps Your Business

Excess of everything is harmful, and it stands true for data as well. While organizations are finding meaningful ways to leverage mountains of data; bad or poor-quality data is creating equal and hindrances. Staying organized or keeping your data healthy and running is critical to business. Duplicate, outdated, inaccurate, incomplete or inconsistent data can have grave impact on your efforts to grow profitably.

69% companies claim that their bad data hinders them providing an enriched customer experience.About 29% organizations believe that their data in accurate and the biggest culprit in this case is manual errors.89% of businesses struggle with managing data quality.

Why your company should cleanse dirty data?

Data is the new oil, but we are living in an era of data explosion or data disaggregation. And data accuracy is the life line for your business. For example, Ola or Uber are required to optimize their fleet program by charting out routes for drivers. Their routing algorithm should be accurate with customer details. But if their customer details are not proper, drivers won’t be able to reach the location in stipulated times and it will result in late pickup and deliveries; along with unnecessary efforts, costs and time waste. Quality data always ...

Read More on Datafloq

Key Takeaways from Google's Meena Chatbot

This article has been penned by Swapan Rajdev, Co-Founder & CTO of Haptik

Chatbots and virtual assistants have been in the news in a big way for the past few days. The reason? One word: Meena.

In a research paper titled Towards an Open Domain Chatbot, Google presented Meena, a “conversational agent that can chat about…anything”.

According to the Google researchers’ who worked on the project, Meena stands in contrast to the majority of current chatbots, that tend to be highly specialized within a particular domain, and perform well as long as users do not stray too much from expected usage. Open-domain chatbots, on the other hand, theoretically have the ability to converse with a user about anything they want.

In practice, however, most current open-domain chatbots often simply do not make sense. At worst, they say things that are inconsistent with what has been said previously, or which indicate a lack of common sense or basic knowledge about the world. At best, they can say something like “I don’t know” – a sensible response to any query, but one which does not address the specific needs of the user.

This is the gap that Google aims to address with Meena, which they claim has ...

Read More on Datafloq

Top 10 AI Trends To Watch Out in 2020

Global spending on cognitive and AI systems will reach $57.6 billion in 2021, as per different industries’ market share. Looking ahead at 2020, enterprises become laser-focused on AI value and leap out of experimentation mode, and ground themselves in reality to accelerate adoption.Here are the top 10 trends in AI that are likely to continue or emerge in the year 2020: ...

Read More on Datafloq

How Big Data Is Impacting the Real Estate Industry

Traditionally, real estate prices have been determined mostly by the number of people moving in and out of an area, along with the availability of jobs, shops, and other opportunities for families. These factors then were summed in simple mathematical models and used by real estate agents to decide home values. But in reality, it’s a mix of factors that work in delicate balance with each other to determine trends in real estate markets.

Technology and software development companies have recognized the complexity of the real estate market and seen it as an opportunity to bring innovative solutions to one of the oldest markets in the world. In the short amount of time that these data-based companies have entered the real estate industry, they’ve fundamentally changed how market analysis is performed.

Here, you’ll find some of the key ways that big data - and the companies processing this data - are impacting the real estate industry.

Micro-trends Shaping Local Markets

One of the biggest impacts that big data analytics has had on the real estate industry is the exposure of micro-trends to a broad audience. The real estate industry has traditionally assumed that neighborhoods are generally the same throughout - even when a neighborhood ...

Read More on Datafloq

Cyber Security Attacks To Watch Out For in 2020

Cyber security teams should stay at the front foot to protect their company data and IT structure. This means watching out for all new mutations and similar attacks is important for organizations. With cyber-attacks constantly increasing, let’s have a look at the latest attacks that organizations and their cyber security testing teams should look out for:More Cloud, More HacksThe cloud is the most common platform for companies to store their data while utilizing their large capacities. However, most of the latest cyber-attacks are increasingly targeting the cloud. Individual applications are a bigger target for hackers. Data breaches are expected to directly target applications that have access to the database. In public cloud environments, new cyber-attacks will gain access to data while decrypted. Weaponized 5G Technology As 5G technology is used in our daily lives, some experts predict that the network could be used by cyber-criminals as civilians mobile users. The advent of 5G may be a catalyst for the development of functional swarm-based attacks. Hackers can create local, ad hoc networks to share information and applications quickly. Swarm Technology It refers to the decentralized systems that use automation to operate without human intervention such as AI and ML. swarm technology ...

Read More on Datafloq

Effectively Distribute Workload to Remote Employees

Remote employees are not like their office-based counterparts. Unlike those at the office, remote workers are less supervised and are flexible in their time. Such “freedom” can result in issues such as unfair workload distribution or failure to meet company expectations. To prevent such occurrences, try tools and techniques for a more comprehensive and in-depth understanding of project management processes. As a beginner trying to acquire a basic knowledge of how to manage remote employees, try the following tips:

Photo Courtesy of freestocks.org via Unsplash1. Install workload management toolsWorkload planners are among the essentials for any business, whether it’s for remote work or for a physical company. However, you need to determine which apps or software you need. For example, you can go for casual communication tools such as Slack and Skype. For workload distribution, you can pick project trackers, like Asana or Trello. Once you’ve picked the workload apps that you need, evaluate if it is the right app or software for a particular task. For example, you need to give a lecture on employee effectiveness. For this scenario, you need an app that ...

Read More on Datafloq

Data on Maternal Mortality Hasn’t Changed. Black Women Fare the Worst

The number of infants that die before their first birthday is still a pressing concern in America. Infant mortality is on the decline, but more can be done to prevent infant deaths – especially among black women.Nearly 300,000 women died from needless complications during pregnancy and childbirth in 2017, according to the World Health Organization (WHO). Most of those deaths occurred in developing countries. Globally, governments and public health agencies have made maternal mortality a priority as part of their work to improve the well-being and the quality of life of constituents.American Infant Mortality by the NumbersInfant mortality occurs when a child passes away before their first birthday. Researchers measure the infant mortality rate by recording the number of infant deaths compared to every 1,000 live births.This metric gives physicians and other healthcare stakeholders vital data about the health of mothers and their babies. It’s a measure of the overall health of the population. In the United States, the infant mortality rate was 5.8 deaths for every 1,000 live births. For black infants, however, that number was nearly twice as high, coming in at and infant mortality rate of 11.3 deaths for every 1,000 live births.Healthy People is an organization ...

Read More on Datafloq

How Data Cleansing Helps Your Business

Leveraging Big Data for Hyper-Personalization

Hyper-personalization is an extension of traditional personalization techniques used in marketing. For example, using a customer’s name in an email or sending a coupon on their birthday. While traditional personalization only used limited customer data, hyper-personalization is based on a wide variety of provided and collected data.

This data often includes demographic, geographic, purchase history, search history, and real-time behavioral information. This data is combined to form individual customer profiles. Marketers can use these profiles to customize product recommendations, time engagement attempts, and adapt service to customer expectations.

In this article, you’ll learn how hyper-personalization works, how big data factors in, and how you can leverage big data and hyper-personalization.

When creating a hyper-personalization framework, you should follow these steps:

Collect your data—you need to consistently collect a wide variety of data on as many of your users as possible. The more data you have on an individual, the easier it is to create personalized content for them. You can collect this data from a variety of sources, including personal profiles and histories, website analytics, and third-party data collection agencies.

Segment your customers—while hyper-personalization doesn’t use segmentation as traditional methods do, it can still be helpful. Segmentation enables you to better visualize your various markets ...

Read More on Datafloq

Wednesday, 26 February 2020

How Big Data Is Impacting the Real Estate Industry

Key Takeaways from Google's Meena Chatbot

Top 10 AI Trends To Watch Out in 2020

Tuesday, 25 February 2020

Validating JCasC configuration files using Visual Studio Code

Configuration-as-code plugin

Problem Statement: Convert the existing schema validation workflow from the current scripting language in the Jenkins Configuration as Code Plugin to a Java based rewrite thereby enhancing its readablity and testability supported by a testing framework for the same. Enhance developer experience by developing a VSCode Plugin to facilitate autocompletion and validation which would help the developer write correct yaml files before application to a Jenkins Instance.

The Configuration as Code plugin has been designed as an opinionated way to configure Jenkins based on human-readable declarative configuration files. Writing such a file should be feasible without being a Jenkins expert, just translating into code a configuration process one is used to executing in the web UI. The plugin uses a schema to verify the files being applied to the Jenkins instance.

With the new JSON Schema being enabled developers can now test their yaml file against it. The schema checks the descriptors i.e. configuration that can be applied to a plugin or Jenkins core, the correct type is used and help text is provided in some cases. VSCode allows us to test out the schema right out of the box with some modifications. This project was built as part of the Community Bridge initiative which is a platform created by the Linux Foundation to empower developers — and the individuals and companies who support them — to advance sustainability, security, and diversity in open source technology. You can take a look at the Jenkins Community Bridge Project Page

Steps to Enable the Schema Validation

a) The first step includes installing the JCasC Plugin for Visual Studio Code and opening up the extension via the extension list. Shortcut for opening the extension list in VSCode editor using Ctrl + Shift + X.

b) In order to enable validation we need to include it in the workspace settings. Navigate to File and then Preference and then Settings. Inside settings search for json and inside settings.json include the following configuration.

{
"yaml.schemas": {
        "schema.json": "y[a]?ml"
    }
}

You can specify a glob pattern as the value for schema.json which is the file name for the schema. This would apply the schema to all yaml files. eg: .[y[a]?ml]

c) The following tasks can be done using VSCode:

a) Auto completion (Ctrl + Space):
  Auto completes on all commands.

b) Document Outlining (Ctrl + Shift + O):
Provides the document outlining of all completed nodes in the file.

d) Create a new file under the work directory called jenkins.yml. For example consider the following contents for the file:

jenkins:
  systemMessage: “Hello World”
  numExecutors: 2

The above yaml file is valid according to the schema and vscode should provide you with validation and autocompletion for the same.

Screenshots

vscode userDocs1 userDocs2

We are holding an online meetup on the 26th February regarding this plugin and how you could use it to validate your YAML configuration files. For any suggestions or dicussions regarding the schema feel free to join our gitter channel. Issues can be created on Github.

The Outsourcers’ Guide to Quality

Data quality is vital when creating reliable algorithms. For companies looking for a solution that equals the quality of their in-house team, it can seem as though outsourcing is an impossible option. Have no fear. The “Outsourcers’ Guide to Quality” will help you predict the level of quality you can expect from a data labeling provider. Download the eBook, today.

Email*
Name:*
First Name: Last Name:
Job Title:*
Function*
Company:*
Street Address:*
Street Address 2:
City:*
State/Province*
Postal Code/Zip Code:*
Country:*
Contact Phone:*
Industry*
Number of Agents*
Number of Employees*

Scrabble Chinese Room and AI Understanding

By Lance Eliot, the AI Trends Insider

If you are a Scrabble fan, you might remember the headlines in 2015 that blared that the winner of the French Scrabble World Championship was someone that did not understand a word of French.

Sacrebleu!

Note that I spelled this stereotypical French phrase as it is spelled in the French language, as one word, rather than the Americanized version of two words with the accent (sacre bleu), which would be important if I was playing Scrabble right now.

Essentially, the word or phrase is an outdated and hackneyed curse that was never particularly used by the French, but crept into the English language and became employed for formulaic portrayals in movies and TV shows.

In any case, let’s focus on the aspect that the winner of the World Champion for the Francophone Classic Scrabble in 2015 was a non-French speaking contestant.

This feat seemed to be nearly impossible.

How could anyone manage to win in Scrabble, a board game dependent upon words, and yet not understand the words being used in this famous and popular sport?

Bizarre, some said.

A miracle, others stated.

I’d say it is nothing more than a magician pulling a rabbit out of a hat or finding your chosen card out of a deck of cards.

Let’s unpack what it means to play Scrabble and see how this winner was able to succeed.

The Inner Game Of Scrabble

In Scrabble, there is a board consisting of squares arranged in a 15 by 15 grid.

Players have various tiles of letters and are supposed to lay down the tiles in a manner that spells out a word.

This can only be done by putting the tiles in a left-to-right or downward manner, meaning you cannot place words in a diagonal or written backwards. There are points scored per tile placed onto the board. The board itself also has squares that when used will amplify the points scored.

What makes the game particularly challenging is that there is a limited set of letters, plus you need to build your word off of a word played on the board (other than at the start), there is a bag of the letters from which you draw your subset of letters, and a slew of other complicating factors come to play.

The play of the game alternates between each of the players.

During your turn, you can play out some or all your tiles if there is a word that you can make, or you can pass but this means that you are giving up that turn and won’t get any points, or you can do an exchange of your subset of letters with whatever remains in the bag and as randomly selected out of the bag.

When I used to play Scrabble with my children, they at first were eager to make a word whenever they could see that it was possible based on their tiles in-hand and what was available on the board. They quickly realized that the problem of impulsively wanting to make words is that you might be setting up your opponent to subsequently score points. Soon enough, the kids soon realized that they needed to try to anticipate whether their opponent could make a word, and attempt to keep their opponent from doing so, by being mindful of the words they were making on the board.

I liked playing the Scrabble game with my kids because it led to discussions, sometimes debates, regarding whether a word was a real word or a made-up word.

You see, playing Scrabble involves first deciding what definitive source will be used to dictate what is a word versus what is not a legitimate word. The kids might have had their own vocabulary from the playground of made-up words, like “sheez-la-cheese,” but I explained that we’d instead use words that were only found in a valid dictionary.

So, we’d grab an English dictionary from our bookshelf and have it at the ready, using it to look up words and verify that they were valid. Even if I already knew a word that was considered in contention, I was happy and eager to see them looking up the word anyway. I figured this would be a means to boost their vocabulary.

Besides considering how the word was correctly spelled, I typically inquired as to the meaning of the word. I did so in hopes that the word would become enmeshed in their minds. If the word was merely a series of letters that happened to make a word, and yet if they did not know what the word meant, I figured it wouldn’t do them much good. When it came time for them to take tests at school and write narratives, I wanted to ensure they knew the nature of the word and could use it in a sensible manner.

This last aspect about understanding the meaning of words is crucial to the story about the non-French speaking winner of the Francophone Scrabble Championship.

In Scrabble, there is no requirement that you actually understand the word that you are spelling out on the board.

You don’t have to state what the word means.

The word merely has to be a valid word.

If you perchance have heard a word and know how it is spelled, or seen it written someplace, and yet if you have no clue what it means, you are perfectly Okay to use it during Scrabble. Nobody is going to ask you to explain the word or use it in a sentence, since that’s not in the official rules of the game (though, when I played Scrabble with my kids, I added that as a rule, sneakily to get them to understand the words and expand their comprehension and vocabulary at the same time).

The non-French speaking contestant had done something that was impressive, he had memorized all the words in the officially used French dictionary, doing so by only memorizing how the words were spelled.

He happened to have a photographic memory capability and was able in nine weeks to memorize the words.

He did not know what the French words meant.

He could not pronounce them per se, since he hadn’t studied the verbalized versions of the words, though I’m sure he could have guessed at how to say many of the words. In that manner, it is perhaps a stretch to suggest that he was a non-French speaking person, due to the aspect that he had memorized French words and likely could try to utter them. He likely could also guess at many of the words in terms of their meanings, since French and English have many of the same underlying roots and bases.

In any case, it seems relatively fair to assert that he wasn’t French speaking since he could not use the words in any fluent manner and had no understanding of the words, along with no grasp of how to form sentences and abide by the semantics of the language. He did though have to learn to count in French from one to ten, in order to participate in the Scrabble game, a requirement of the contestants.

I’ve now revealed how the magician pulled off the magic act.

Similar to describing how a rabbit got into that hat of the magician, or how your card was marked or planted into a deck of cards, the secret in this case of Scrabble is that you don’t need to understand the words and merely need to know how to spell them. Admittedly, memorizing an entire dictionary of words is somewhat impressive, though having a photographic memory makes it relatively “easy” to do.

To him, the words were essentially icons or images.

Sure, you ultimately need to discern each separate letter that is in a given word, but you can pretty much just remember what the word looks like and then have it ready when needed.

Pretend that letters are only scratches that consist of lines and curves. Those various lines and curves make letters, and the letters are placed next to each other to make words. It is a primitive way to consider the nature of words and letters, though quite effective and the only necessity for playing Scrabble. They are nothing more than blobs.

Upon hearing about this contestant winning, I was immediately aware that he would not have had to “understand” the French language to win at such a Scrabble tournament.

Thus, I was not especially surprised or taken aback.

My first thought was that there is actually a lot more to Scrabble beyond memorizing patterns of letters and words.

More Twists To Scrabble

Being smart about the game play is essential in Scrabble, and especially at any vaunted tournament.

The strategies and tactics that you use in Scrabble are crucial to winning. You cannot just take anyone that happens to have a photographic memory and have them winning Scrabble contests all around the planet. It is like playing Poker, namely that being able to play by the rules and realize what the different cards in the deck represent won’t let you win those million-dollar Las Vegas gambling contests. You need to have a ton of game-playing skills and hone them to be able to play at the top-level of competition.

It turns out that the winner of the Francophone Scrabble Championship was a five-time winner of the North American Scrabble Championships and a three-time winner of the World Scrabble Championships.

All of those competitions were in English.

Regardless though of the language used in those competitions, the fact that he had won those contests demonstrated that he knew how to play the Scrabble game and must have finely tuned his strategies and tactics for it.

In that way, he was able to deploy his Scrabble playing expertise into the context of the French version, since it is still the same fundamental game. By memorizing the French words, he had put together a potent combination, consisting of his highly honed Scrabble game playing strategies and tactics, along with having at his fingertips (in his mind) an entire dictionary of allowed words. It was a kind of double-whammy that likely made things tough on his French-speaking competition.

One wonders how many of the other contestants had a photographic memory and had memorized as many words as he had?

Probably not many of the contestants have that knack. Even if there were other contestants with a similarly sized word set in their minds, you then have the aspect of Scrabble game playing strategies and tactics. So, he might have bested some of them in that manner.

There is also the role of chance involved in the game, since you don’t know beforehand what letters you will get.

There is the randomness of drawing tiles (letters) from the bag. Presumably, if you play enough games, over time the “luck” or “unluck” of your draws will even out and the players will then be winning based on their actual game play expertise, though this is only likely if the number of games played is sufficiently large. Scrabble competitions try to deal with this matter by having multiple games between players, but in-the-small it is not necessarily the case that the luck factor is going to be expunged.

Another facet of the Scrabble game is the somewhat false assumption that by playing words with the largest number of letters that you’ll be able to prevail in terms of getting the highest total score by the end of the game. If you play a bunch of rounds, you’ll learn soon enough that the largest words also tend to offer ripe opportunities for your competition. In fact, some studies have suggested that you are likely better off in using predominantly four-letter and five-letter words, assuming that you are playing strongly and that your opponent is also a strong player too.

Bringing up the topic of Scrabble will often elicit a smile from AI developers and they’ll likely ask or point out gently that “didn’t we solve that already” with AI?

This makes me cringe somewhat because it is a bit of an overstatement.

AI Playing Scrabble

Yes, there are some quite famous AI programs that do play Scrabble well.

The most historically notable ones are likely Maven and Quackle.

Maven was first developed around the mid-1980s and became the star around which other offshoots tended to appear. The structure of Maven’s approach consists of dividing a Scrabble match into a mid-game, a pre-endgame, and an endgame set of phases (the mid-game is somewhat a misnomer since it also serves as the start-game capability too).

During the mid-game portion, the AI of Maven is ascertaining all possible plays based on the tiles in the rack of the player and what’s on the board and uses relatively simple rules or heuristics to try to figure out which of the valid words based on its rack might be most prudent to play. There is a simulation or “simming” done to try to look-ahead at various moves and countermoves, though in the initial incarnations it was only a two-ahead look (a 2-ply deep). This is considered a truncated version of the Monte Carlo simulation and not a full-bodied MCTS (Monte Carlo Tree Search) implementation.

Other variants of Maven included the use of a DAWG (Directed Acyclic Word Graph), which tends to run fast and doesn’t require an elaborate algorithm per se, and latter used the GADDAG (this naming was intended to be smarmy, it is the letters DAG for Directed Acyclic, spelled backwards and then forwards).

The end-game is a different kind of challenge and kicks-in once the bag of letters is empty.

This means that there is no longer a random draw of letters. You might therefore assume that things are pretty simplified, since you then know all the letters already on the board, you know the letters in the racks, and so you presumably can handle a perfect information situation, which in the case of Maven the B-star search was utilized. Part of the difficulty is there is usually a time limit involved and the search space can become large and computationally expensive in terms of time consumed.

Quackle came along after Maven and employs many similar game playing approaches, along with a few other nuances. If you are interested in Scrabble AI game play, the Quackle is readily available as open source and can be found in places such as GitHub.

Both Maven and Quackle have had circumstances wherein they were used to compete against topnotch human Scrabble players.

Though they have had some impressive wins, it does not mean that they have “solved” the playing of Scrabble by AI. I emphasize this because of the sometimes smirks that I get from AI developers that believe there is nothing left to do in the Scrabble game regarding trying to use AI. Anyone that says this is either unaware of the reality of AI Scrabble game playing, or they assume that if there were some wins by AI Scrabble game playing system that it implies the matter is completed and no further effort would be worthwhile.

Somewhat similar to the non-French speaking human winner of the Francophone Scrabble Championship, there is an added edge in this particular kind of game if you can have at the ready an entire dictionary of words.

Any human player that cannot commit to their own mental memory an entire dictionary of words is obviously at a disadvantage.

It is not necessarily an insurmountable disadvantage since, as I’ve already mentioned, simply knowing about the spelling of all the possible words is not all that it takes to play the game well. You could have memorized all words in the dictionary and still lose a match due to inadequate strategy. You could even have all those words memorized and play a topnotch strategy, and still lose due to the skills of your opponent and/or due to the luck-of-the-draw in terms of the letters being randomly drawn from the bag.

There is also the time factor involved.

A player that can assess more possibilities in the length of time allowed per move is presumably going to have a greater chance of making a better move than otherwise if they could not examine as many options. This limit applies to the human player and their mental processing, and likewise to the AI and its use of computer cycles for processing.

Of course, the depth of mental processing is not necessarily the winning approach since it could be that there are lots of possibilities that aren’t worth the mental effort, and nor time, when figuring out your next move.

In short, just because the computer can have at-the-ready an entire dictionary of words does not ergo mean it is going to win. Likewise, even if the AI has an algorithm that uses all kinds of short-cuts and statistics to try and ascertain the seemingly most prudent choice, there is still room for improvement in those algorithms.

This is not a done deal and should not be construed as such.

When considering Scrabble, we might also want to take into account the role of “understanding” when it comes to playing this popular game.

I’ve already indicated that the non-French speaking winner did not “understand” the words that he was using while playing the French version of Scrabble. Overall, he had no idea what those words meant. They were scratches of lines and curves. These words were icons or images. They were blobs.

That’s a good match for using a computer system since the computer and the AI do not “understand” things in the way that we assume humans do.

Meaning Of Understanding Is A Key Matter

In playing Scrabble, any player, whether human or AI, does not need to “understand” the words since those are only being used as objects. Any circumstance involving long lists of objects is likely to give the computer a potential advantage since it can presumably have those in computer memory whereas a human is less likely to be able to do so in their own mind. Having a photographic memory by a human would certainly be an exception, though we need to realize there aren’t many humans that seem to have a photographic memory.

Now that we’ve carved out any need for “understanding” in terms of the dictionary of words used in Scrabble, we need to acknowledge the perhaps hidden form of “understanding” needed during the playing of the game. The strategies and tactics used would be applicable to what we commonly refer to as having an “understanding” of something.

We don’t know for sure what goes on in the heads of a Scrabble player and can only guess at what they might be thinking during the playing of a game.

You can of course ask a Scrabble player what they were thinking. They will tell you what they believe they were thinking. We don’t know that it is the same thing as what they were really thinking. It could be a made-up rationalization. If you ask me what I was thinking about during a Scrabble game, and if I don’t want you to believe that I was playing the game by some oddball means, I might tell you that I carefully examined the board, I mentally calculated the points, and I thoughtfully determined my next move. I could sincerely “believe” that’s what my mind was doing.

We don’t know that to be the case. Your mind might be using some other approach entirely. It might seem logical the way you describe it, but that doesn’t make it so.

The AI algorithms and techniques employed in the Scrabble playing of Maven and Quackle are maybe similar to what happens in the human mind or maybe not. I’d dare say, most likely probably not. We have come up with some fascinating mathematical and computational approaches that appear to be useful and can compete against humans in a game such as Scrabble.

Does this mean that those AI systems “understand” the game of Scrabble? You’d be hard pressed to say yes.

Revisiting The Chinese Room Argument

This is reminiscent of the famous Chinese Room argument.

For anyone involved in AI, you ought to be familiar with the thought experiment known as the Chinese Room.

It goes like this. We develop something we regard as AI which we’ll place into a room and that can take-in Chinese characters as input and will emit Chinese characters as output, doing so in a manner that a human that is feeding the Chinese characters as input and is reading the Chinese characters of output is led to believe that the AI is a human being. In that sense, this AI passes the infamous Turing Test.

The Turing Test is the notion that if you have a computer and a human, and another human asks questions of the two, when the inquiring human cannot differentiate the computer versus the human, the computer is considered as having passed the Turing Test. It therefore would seem that the computer is able to express intelligence as a human can.

For my review and assessment of the Turing Test, see: https://www.aitrends.com/selfdrivingcars/turing-test-ai-self-driving-cars/

Is the AI that’s inside that Chinese room able to “understand” in the same manner that we ascribe the notion of being able to “understand” things as people do?

You could ask that same question of the Turing Test, but the twist somewhat with the Chinese Room is the added element that I will describe next.

Suppose we put an actual human into this Chinese Room. They do not understand a word of Chinese. We also give to the human the same computer program that embodies the AI system. This human endeavors to do exactly what the computer program does, following each instruction explicitly, perhaps using paper and pencil to do so. Notice that the AI is not going to be doing the processing per se, and instead the human inside the Chinese Room will be doing so, following carefully step-by-step whatever the AI would have done.

Presumably, the human inside the Chinese Room is going to once again be able to take-in the Chinese characters as input and emit Chinese characters as output, which we assume will occur due to abiding strictly by the steps of the already-successful AI and be able to convince the human outside the room that the room contains intelligence. The human in the Chinese room does not understand a word of Chinese, and yet has been able to respond to a Chinese inquirer as though they did understand Chinese, even though it was a “trick” because the human merely followed “mindlessly” the steps indicated of the AI program.

It is claimed that this showcases that there was no real sense of “understanding” involved by the AI and nor by the human that was inside the Chinese room.

Some define the possibility of “strong AI” to be AI that does have a sense of “understanding,” while so-called “weak AI” does not and is merely some kind of simulated version of what we refer to as a sense of understanding. The Chinese Room thought experiment is intended to highlight the nature of “weak AI” and do so by way of illustration (which simultaneously also highlights what we consider to not be “strong AI”).

Readers should be aware that not everyone accepts the definitions of weak AI and strong AI in this manner. For example, some would say that weak AI is an AI system that might be brittle and easily fooled or confused, while strong AI is an AI system that is more robust and hardier. I hope it is apparent that the use of “weak AI” and “strong AI” in the context of the Chinese Room is quite a different matter of how that vocabulary is used.

A philosopher named John Searle proposed the Chinese Room thought experiment, doing so in 1980, and ever since then there has been quite a response to it. There are lots of arguments about alleged loopholes and fallacies in the thought experiment and this Chinese Room notion. Some critics decry the Chinese Room. Whether you refute it, love it, hate it, despise it, or even believe it is a waste of time, or believe it is a hallmark of thinking about thinking, it has become a longstanding point of discussion and some would consider it a classic of cognitive science and of AI.

I’m not going to tackle the Chinese Room aspects herein. Instead, I bring it up to highlight my earlier point about the playing of Scrabble. I had indicated that it is unknown as to what it means to have “understanding” when it comes to the strategies and tactics of playing Scrabble. We can put to the side any sense of “understanding” about the words used in the Scrabble game, since those are merely objects and in that manner we could claim they are minimal in terms of having to “understand” what they are.

But what about the Scrabble game playing?

The AI program of Maven and Quackle, do they embody a sense of “understanding” about the playing of Scrabble, akin to when a human has “understanding” as they play the game?

Most would agree that those AI programs do not have any “understanding” in them.

They are the same as the Chinese Room.

Role Of Machine Learning And Deep Learning

You might be wondering whether Machine Learning or Deep Learning could maybe rescue us in this situation.

Typically, a Machine Learning or Deep Learning approach involves the use of a large-scale artificial neural network. It is somewhat based on the same aspects of how the human brain perhaps operates, incorporating the use of neurons, synapses, and so on. Today’s artificial neural networks are a far cry of being anything close to what happens in the wetware, the human brain. As such, it is at best a simplistic simulation of those biological and biochemical aspects of the brain.

In any case, the assumption and future hope is that if we can keep making computer-based artificial neural networks more and more akin to the human brain, possibly we will have human intelligence emerge in these artificial neural networks. Maybe it won’t happen all at once and instead appear in dribs and drabs. Maybe it won’t ever appear. Maybe there is a secret sauce of the operation of the brain that we’ll never be able to crack open. Who knows?

There haven’t been many attempts to play Scrabble via the use of an artificial neural network.

The more straight-ahead methods of using various AI search space techniques and algorithms has been the predominant approach used. It seems to make sense that you would use these more overt or symbolic types of approaches, doing a direct kind of programming to solve the problem, rather than using a neural network, which is more of a bottoms-up approach rather than a top-down approach.

With an artificial neural network, it’s not quite clear how to best train the neural network for the Scrabble game. Usually, you feed tons of examples or in this case game plays, and the attempt to train the neural network to how the game is played. This in a sense provides a mathematical means to have the artificial neural network do pattern matching and “discover” in a numeric way the strategies and tactics played. This approach has been used in other games such as chess.

If you ponder the difference between a game like chess and a game like Scrabble, you’d readily notice some key attributes that make them very different. In chess, all the playing pieces are known and placed on the board at the start of the game. In the case of Scrabble, the letters are hidden in a bag and you are dealt out a subset at a time, therefore you have imperfect information and you are also going to be dependent upon random chance of what will occur during the game.

Collecting together a massive number of chess games and being able to feed those as data into an artificial neural network is somewhat easy task to be undertaken. Doing the same for Scrabble games is not so easily done. Even if you do this, the idea of pattern matching based on those games is going to be quite unlike the pattern matching of a chess game.

Here’s the rub.

If you believe that the use of Machine Learning or Deep Learning is our best shot at achieving human intelligence via AI, presumably we should be using Machine Learning or Deep Learning on trying to craft better and better Scrabble playing automation.

At this time, it would seem that our progress on Machine Learning or Deep Learning is not far enough along to merit believing that the existing employment of Machine Learning or Deep Learning (as we know if it today) would surpass the more direct and programmatic versions of AI such as Maven and Quackle. Perhaps at some future time, this will shift toward the Machine Learning or Deep Learning side of things.

Here’s another thought to consider.

Are the Machine Learning and Deep Learning systems of today able to “understand” in the same manner that we assume that humans can “understand” things?

You’d be hard pressed to have any reasonable AI developer say yes.

If that’s the case that these Machine Learning and Deep Learning systems of today are not able to “understand” (in a human sense of “understanding”), will they at some future point be able to do so? Will it be because they become so large-scale in size that “understanding” arises out of the sheer magnitude? Or, will we be doing something else with these models that takes them closer and closer to the true wetware of the human brain?

For those that believe an AI singularity is coming, see my article: https://www.aitrends.com/selfdrivingcars/singularity-and-ai-self-driving-cars/

For the potential dangers of super-intelligent AI, see my article: https://www.aitrends.com/selfdrivingcars/super-intelligent-ai-paperclip-maximizer-conundrum-and-ai-self-driving-cars/

For my article about whether AI might be a Frankenstein, see: https://www.aitrends.com/selfdrivingcars/frankenstein-and-ai-self-driving-cars/

For my article about Deep Learning and plasticity, see: https://www.aitrends.com/ai-insider/plasticity-in-deep-learning-dynamic-adaptations-for-ai-self-driving-cars/

AI Self-Driving Cars And Scrabble

What does this have to do with AI self-driving driverless autonomous cars?

At the Cybernetic AI Self-Driving Car Institute, we are developing AI software for self-driving cars. One aspect that is not widely realized involves the lack of “understanding” that the AI of self-driving cars of today embody and whether that poses safety and risks that aren’t being well-discussed.

Allow me to elaborate.

I’d like to first clarify and introduce the notion that there are varying levels of AI self-driving cars. The topmost level is considered Level 5. A Level 5 self-driving car is one that is being driven by the AI and there is no human driver involved. For the design of Level 5 self-driving cars, the automakers are even removing the gas pedal, brake pedal, and steering wheel, since those are contraptions used by human drivers. The Level 5 self-driving car is not being driven by a human and nor is there an expectation that a human driver will be present in the self-driving car. It’s all on the shoulders of the AI to drive the car.

For self-driving cars less than a Level 5 and Level 4, there must be a human driver present in the car. The human driver is currently considered the responsible party for the acts of the car. The AI and the human driver are co-sharing the driving task. In spite of this co-sharing, the human is supposed to remain fully immersed into the driving task and be ready at all times to perform the driving task. I’ve repeatedly warned about the dangers of this co-sharing arrangement and predicted it will produce many untoward results.

For my overall framework about AI self-driving cars, see my article: https://aitrends.com/selfdrivingcars/framework-ai-self-driving-driverless-cars-big-picture/

For the levels of self-driving cars, see my article: https://aitrends.com/selfdrivingcars/richter-scale-levels-self-driving-cars/

For why AI Level 5 self-driving cars are like a moonshot, see my article: https://aitrends.com/selfdrivingcars/self-driving-car-mother-ai-projects-moonshot/

For the dangers of co-sharing the driving task, see my article: https://aitrends.com/selfdrivingcars/human-back-up-drivers-for-ai-self-driving-cars/

Let’s focus herein on the true Level 5 self-driving car. Much of the comments apply to the less than Level 5 and Level 4 self-driving cars too, but the fully autonomous AI self-driving car will receive the most attention in this discussion.

Here’s the usual steps involved in the AI driving task:

Sensor data collection and interpretation
Sensor fusion
Virtual world model updating
AI action planning
Car controls command issuance

Another key aspect of AI self-driving cars is that they will be driving on our roadways in the midst of human driven cars too. There are some pundits of AI self-driving cars that continually refer to a Utopian world in which there are only AI self-driving cars on the public roads. Currently there are about 250+ million conventional cars in the United States alone, and those cars are not going to magically disappear or become true Level 5 AI self-driving cars overnight.

Indeed, the use of human driven cars will last for many years, likely many decades, and the advent of AI self-driving cars will occur while there are still human driven cars on the roads. This is a crucial point since this means that the AI of self-driving cars needs to be able to contend with not just other AI self-driving cars, but also contend with human driven cars. It is easy to envision a simplistic and rather unrealistic world in which all AI self-driving cars are politely interacting with each other and being civil about roadway interactions. That’s not what is going to be happening for the foreseeable future. AI self-driving cars and human driven cars will need to be able to cope with each other.

For my article about the grand convergence that has led us to this moment in time, see: https://aitrends.com/selfdrivingcars/grand-convergence-explains-rise-self-driving-cars/

See my article about the ethical dilemmas facing AI self-driving cars: https://aitrends.com/selfdrivingcars/ethically-ambiguous-self-driving-cars/

For potential regulations about AI self-driving cars, see my article: https://aitrends.com/selfdrivingcars/assessing-federal-regulations-self-driving-cars-house-bill-passed/

For my predictions about AI self-driving cars for the 2020s, 2030s, and 2040s, see my article: https://aitrends.com/selfdrivingcars/gen-z-and-the-fate-of-ai-self-driving-cars/

Returning to the topic at-hand, I’ve been discussing the nature of Scrabble and how humans and how AI systems embody or do not embody a sense of “understanding” in the meaning of what we believe humans can think about things.

When a human drives a car, do you believe that the human is employing “understanding” in some manner, such as understanding how a car operates, understanding how traffic flows and cars maneuver in traffic, and how humans drive cars, and how humans as pedestrians act when near cars, etc.?

If you say yes, this next question is then prompted by the Scrabble discussion and the Chinese Room discussion, namely, will the AI of self-driving cars need to also embody a similar sense of “understanding” in order to properly, safely, and appropriately be driving cars on our public roadways?

Yes or no?

Caught you!

I say that I caught you because if you say yes, and you are of the belief that the AI of self-driving cars needs to have a sense of “understanding” about driving as humans do, right now the auto makers and tech firms are not anywhere close to achieving “understanding” in these AI systems. Simply stated, the AI of today’s and even near-future AI self-driving cars do not embody “understanding” at all.

The AI of today’s and the near-future self-driving cars is akin to the Scrabble game AI.

By-and-large, most of the AI being used in an AI self-driving car is the programmatic type that uses various AI techniques and algorithms, but it is not what we would reasonably agree is any kind of “understanding” that is going on.

You might right away be claiming that since the AI of self-driving cars is often making use of Machine Learning and Deep Learning, it suggests that perhaps the AI is getting closer to having “understanding” in the manner that deep artificial neural networks might someday invoke.

Problematically, those neural networks of today are not yet far advanced toward what someday we all hope might happen with extremely large-scale neural networks and ones that are more closely modeled with the human brain. Furthermore, the neural networks aspects are currently just a small part of the AI stack for self-driving cars.

The use of Deep Learning or Machine Learning is primarily used in the sensors portion of the AI systems for self-driving cars. This makes sense when you consider the duties of the AI subsystems involved in the sensor portion of the driving task. The sensors collect a ton of data. This might be images from the cameras, this might be radar data, LIDAR data, ultrasonic data, and so on.

It is a ready-made situation to use Machine Learning or Deep Learning.

We can for example beforehand collect lots of images of street signs. Those can be used to train an artificial neural network. We can then put into the on-board self-driving car system the runnable neural network that will examine an image of a street scene and hopefully be able to detect where a street sign is, along with classifying what kind of street sign it found, such as a Stop sign or a Caution sign.

For my article about street signs and neural networks, see: https://www.aitrends.com/selfdrivingcars/making-ai-sense-of-road-signs/

For the street scene analyses of Deep Learning, see: https://www.aitrends.com/selfdrivingcars/street-scene-free-space-detection-self-driving-cars-road-ahead/

For my article about the use of probabilities, see: https://www.aitrends.com/ai-insider/probabilistic-reasoning-ai-self-driving-cars/

For safety and AI self-driving cars, see my article: https://www.aitrends.com/selfdrivingcars/safety-and-ai-self-driving-cars-world-safety-summit-on-autonomous-tech/

For my article about common sense reasoning, see: https://www.aitrends.com/selfdrivingcars/common-sense-reasoning-and-ai-self-driving-cars/

Once Again Understanding Rears Its Head

The AI of the self-driving car does not “understand” the street signs, at least not in the manner that we might believe a human has such an understanding.

The street sign is merely an object, akin to the tiles on the Scrabble board of letters that are lines and curves. The rest of the AI has to then use various algorithms and techniques to ascertain what those blobs signify in terms of the action that the self-driving cars should undertake. This would be similar to the Scrabble playing AI that uses various techniques to undertake the strategies and tactics of the game.

As I’ve repeatedly stated in my writings and presentations, the AI of self-driving cars does not have any common-sense reasoning capability. I mention this because many would say that the act of “understanding” must involve having common sense reasoning. If that indeed is an essential and inseparable ingredient for being able to understand, the sad news is that we are very far away from having any kind of truly robust common-sense reasoning systems.

In essence, we are for now going to be foregoing having AI that has any semblance of human “understanding” and furthermore this applies to the AI of self-driving cars.

When I earlier stated that I caught you, my question had been purposely posed to ask whether you thought that AI self-driving cars must have some semblance of human “understanding” to be able to properly and appropriately drive a car on our roadways.

The catch was that if you say yes, well, there then shouldn’t be any AI self-driving cars on our roadways as yet. If you say no to that question, you are then expressing a willingness to have AI that is less-than whatever human “understanding” consists of, and you are suggesting that you are comfortable with that kind of AI being able to drive on our roadways.

This brings me back to another earlier point too. I had mentioned that some AI developers falsely seem to believe that Scrabble has been “solved” as an AI problem. I presume that you now know that though progress has been made, there is still much room to go before we could somehow declare that AI has conquered Scrabble. The aspect that there are in existence some AI programs that can best a human, some of the time, would not seem to be a suitable way to plant a flag and say that the AI that has done so is the best that can be done.

It would hopefully be apparent that I am aiming to say the same thing about the AI for self-driving cars.

We are going to inextricably end-up with this version 1.0 of AI self-driving cars. Let’s assume and hope that they are able to drive on our roadways and do so safely (that’s a loaded word and one that can mean different things to different people!).

Will that mean that we’ve conquered the task of driving a car?

Some might want to say yes, but I beg to differ.

I’m betting that we are going to be able to greatly improve on that version 1.0, and reach a version 2.0, perhaps 3.0, and so on, each getting better and better at driving a car. This will include doing some things that human drivers do, while also doing some things that human drivers do that they ought not to do when driving a car.

For my Top 10 predictions about AI self-driving cars, see: https://www.aitrends.com/ai-insider/top-10-ai-trends-insider-predictions-about-ai-and-ai-self-driving-cars-for-2019/

For the timeline of the advent of AI self-driving cars, see my article: https://www.aitrends.com/selfdrivingcars/gen-z-and-the-fate-of-ai-self-driving-cars/

For my article on the reframing of levels of autonomy and AI self-driving cars, see: https://www.aitrends.com/selfdrivingcars/reframing-ai-levels-for-self-driving-cars-bifurcation-of-autonomy/

For the debate about driving controls and AI self-driving cars, see my article: https://www.aitrends.com/selfdrivingcars/ai-boundaries-and-self-driving-cars-the-driving-controls-debate/

Conclusion

Congratulations to the non-French speaking winner of the French-based Scrabble tournament.

Just to say, I would be offering the same congratulations if it was a non-English speaking French player that was able to win the English-speaking North American tournament.

Winning a Scrabble competition at the topmost level is a feat of incredible strategy and thinking.

I have used the Scrabble aspects as a means to draw your attention to the nature of “understanding” in the matter of human thinking. Per the Chinese Room, we appear to be still at a great distance in today’s AI of reaching to any kind of “understanding” that we might agree exists in humans. Whether you like the Chinese Room exemplar or not, it provides another means to bring up the importance of thinking about thinking and trying to figure out what “understanding” actually entails.

For AI self-driving cars, they are coming along, regardless of AI having not yet cracked the secrets of how to achieve the “understanding” that humans have. We are going to presumably accept the notion that we will have AI systems, minus “understanding” which will be driving around cars on our public roadways.

Can those presumed non-understanding AI systems be proficient enough to warrant driving multi-ton cars that will be making human-related life-and-death decisions at every moment as they zip along our streets and highways?

Time will tell.

Meanwhile, if we do get there, don’t fall into the mental trap that the matter has been solved and that there is no AI left to yet be further attained. I assure you, there will be plenty of AI roadway left to be driven and plenty of opportunity for AI developers and researchers in doing so. Hey, the word “opportunity” is an 11-letter word, I wonder if that will fit during my next Scrabble game.

This content is originally posted on AI Trends.

[Ed. Note: For reader’s interested in Dr. Eliot’s ongoing business analyses about the advent of self-driving cars, see his online Forbes column: https://forbes.com/sites/lanceeliot/]

Cyber Security Attacks To Watch Out For in 2020

We would see a very different banking sector in years to come: RBI governor

Banks would continue to be in a position of primacy in finance with them adopting technologies, RBI chief said.

Monday, 24 February 2020

Introducing Databricks Ingest: Easy and Efficient Data Ingestion from Different Sources into Delta Lake

New Data Ingestion Network for Databricks: The Partner Ecosystem for Applications, Database, and Big Data Integrations into Delta Lake

Organizations have a wealth of information siloed in various sources, and pulling this data together for BI, reporting and machine learning applications is one of the biggest obstacles to realizing business value from data. The data sources vary from operational databases such as Oracle, MySQL, etc. to SaaS applications like Salesforce, Marketo, etc. Ingesting all this data into a central lakehouse is often hard, in many cases requiring custom development and dozens of connectors or APIs that change over time and then break the data loading process. Many companies use disparate data integration tools that require data engineers to write scripts and schedule jobs, schedule triggers and handle job failures, which does not scale and creates massive operational overhead.

Introducing the Data Ingestion Network

To solve this problem, today we launched our Data Ingestion Network that enables an easy and automated way to populate your lakehouse from hundreds of data sources into Delta Lake. We are excited about the many partners announced today that have joined our Data Ingestions Network – Fivetran, Qlik, Infoworks, StreamSets, Syncsort. Their integrations to Data Ingest provide hundreds of application, database, mainframe, file system, and big data system connectors, and enable automation to move that disparate data into an open, scalable lakehouse on Databricks quickly and reliably. Customers using Azure Databricks already benefit from the native integration with Azure Data Factory to ingest data from many sources.

Key Benefits of the Data Ingestion Network

1. Real-time, Automated Data Movement

The ingest process is optimized for change data capture (CDC), and enables easy automation to load new or updated datasets into Delta Lake. Data engineers no longer need to spend time developing this complex logic, or processing the datasets manually each time. The data in Delta Lake can be automatically synced with changes and kept up to date.

2. Out-of-the-Box Connectors

Data engineers, Data scientists, and Data analysts have access to out-of-the-box connectors through the Data Ingest Network of partners to SaaS applications like Salesforce, Marketo, Google Analytics, and Databases like Oracle, MySQL and Teradata, plus file systems, mainframes, and many others. This makes it much easier to set up, configure and maintain the data connections to hundreds of different sources.

3. Data Reliability

Data ingestion into Delta Lake supports ACID transactions that makes data ready to query and analyze. This makes more enterprise data available to BI, reporting, data science and machine learning applications to drive better decision-making and business outcomes.

Data Ingestion Set Up in 3 Steps

End-users can discover and access the integration setup the Data Ingestion Network of partners through the Databricks Partner Gallery.

Step 1: Partner Gallery

Navigate to the Partner Integrations menu to see the Data Ingestion Network of partners. We call this the Partner Gallery. Follow the Set up guide instructions for your chosen partner.

Step 2: Set up Databricks

Next, set up your Databricks workspace to enable partner integrations to push data into Delta Lake. Do the following:

Create a Databricks token that will be used for authentication by the partner product

From the Databricks cluster page, copy the JDBC/ODBC URL

Step 3: Choose the data sources, select Databricks as the destination

Using the partner product, choose the data sources you want to pull data from and choose Databricks as the destination. Enter the token and JDBC information from step 2, and set up the job that will then pull data from your data source and push it into Databricks in the Delta Lake format.

That’s it! Your data is now in Delta Lake, ready to query and analyze.

A Powerful Data Source Ecosystem to Address Data Ingestion Needs

The Data Ingestion Network is a managed offering that allows data teams to copy and sync data from hundreds of data sources using auto-load and auto-update capabilities. Fivetran, Qlik, Infoworks, StreamSets, and Syncsort are available today, along with Azure Data Factory that already provided native integration for Azure Databricks customers to ingest data from many sources. Together these partners enable access to an extensive collection of data sources that are both cloud-based and on-premises.

The Goal of the Data Ingestion Network

With the Data Ingestion Network, we set out to build an ecosystem of data access that allows customers to realize the potential of combining big data and data from cloud-based applications, databases, mainframes and files systems. By simplifying the data ingestion process compared to traditional ETL, customers have the ability to overcome the complexity and maintenance cost typically associated with pulling data together from many disparate sources. This accelerates the path to maximizing the business value from data across BI, reporting and machine learning applications.

To learn more:

Talk to an expert: Contact us

Try Databricks for free. Get started today.

The post New Data Ingestion Network for Databricks: The Partner Ecosystem for Applications, Database, and Big Data Integrations into Delta Lake appeared first on Databricks.