
If you just want to weigh yourself, you can use any old scale on the market.
Data Science, Machine Learning, Natural Language Processing, Text Analysis, Recommendation Engine, R, Python

If you just want to weigh yourself, you can use any old scale on the market.

Whether you want to block out noises from outside your bedroom, calm your mind, or both, there’s a white noise machine for you.
With GDPR enforcement rapidly approaching (May 25, 2018), many companies are still trying to figure out how to comply. A big pain point, particularly for companies who utilize data lakes to store vast amounts of data, is how to comply with one of the main requirements under the GDPR – data subject requests, also known as “DSRs”.
What is a DSR?
One of the most operationally significant parts of the GDPR for companies is the data subject request. The GDPR provides all European data subjects (that is, any individual person located in Europe) with a set of enumerated rights related to their personal data including the right to:
Companies have, unless the request is “complex” or “numerous”, thirty days from receipt of the data subject request to comply with the request (keeping in mind any applicable exceptions).
So what’s the big deal?
Finding data in a data lake is hard; being sure that you’ve found all data about a particular individual is very hard. And many data lakes do not even enable users to perform “delete” operations, even once the data is located, so actually removing it may be practically impossible. In the best case, finding and removing such data is computationally difficult, expensive, and time consuming. And if a company receives more than just a few data subject requests in a short period of time, the resources spent to comply with the requests could be significant. Further, failure to comply with the GDPR could result in significant penalties, potentially as high as €20 million (or even more – up to 4% of a company’s global annual revenues).
So that sounds bad. Is there anything that can be done?
Fortunately, Databricks offers a solution. Enter Databricks Delta, a unified data management system built into the Databricks platform, that brings data reliability and performance optimizations to cloud data lakes.
Databricks Delta’s structured data management system adds transactional capabilities to your data lake that enable you to easily and quickly search, modify, and clean your data using standard SQL DML statements (e.g. DELETE, UPDATE, MERGE INTO). To accomplish this, first ingest your raw data into Delta tables which adds metadata to your files. Once ingested, you can easily search and modify individual records within your Delta tables to meet DSR obligations. The final step is to make Delta your single source of truth by erasing any underlying raw data. This removes any lingering records from your raw data sets. We suggest setting up a retention policy with AWS or Azure of thirty days or less to automatically remove raw data so that no further action is needed to delete the raw data to meet DSR response timelines under the GDPR.
Can you provide an example of how this works?
Let’s say your organization received a DSR to delete information related to Justin Olsson (jdo@databricks.com). After ingesting your raw data into Delta tables, Databricks Delta would enable you to find and delete information related to user jdo@databricks.com by running two commands:
(1) DELETE FROM data WHERE email LIKE 'jdo@databricks.com';
(2) VACUUM data;
The first command identifies records that have the string “jdo@databricks.com” stored in the column email, accounting for varying case (e.g., JDO@databricks.com would also match), and deletes the data containing these records, rewriting the respective underlying files with the user’s data removed. The second command cleans up the Delta table, removing any stale records that have been logically deleted and those that are outside of the default retention period (e.g., 7 days).
After running these commands, and waiting for your default retention period to delete the underlying raw files, you would be able to state that you had removed records relating to the user jdo@databricks.com from your data lake.
Okay, that sounds great , but if I put my data in a Delta table, won’t I be locked in? What if I want to go somewhere else?
Nope! Databricks Delta is architected with portability in mind. Databricks Delta uses an open file format (parquet) and you can at any time (either if you ever decide to stop using Delta or if you need to output data to a system that cannot read Delta tables) quickly and easily convert your data back into a format that can be read by other tools. While doing so, particularly on an ongoing basis, would leave you with the additional DSR obligation of deleting or exporting any personal data that might be contained in the data that was moved out of Databricks Delta, it too will have benefitted from flowing through Databricks Delta, as it will be in a much more structured format, dramatically simplifying that process as well.
Learn more and watch a live demo
Join our webinar, Is Your Data Lake GDPR Ready? How to Avoid Drowning in Data Requests, on May 9 for a live a demo and tips for overcoming the challenges of DSRs in a big data world.
This webinar will cover:
Register to attend this session.
--
Try Databricks for free. Get started today.
The post How to Avoid Drowning in DSRs in a Data Lake appeared first on Databricks.

The brand new Oculus Go headset, announced at Facebook’s F8 conference today, is a self-contained VR headset that doesn’t require…

Interested in trying out a mechanical keyboard but don’t know here to start?

On behalf of the Jenkins GSoC team and mentors, I would like to welcome Shenyu Zheng, Udara De Silva, Pham Vu Tuan and Abhishek Gautam. They will be working on Google Summer of Code projects in the Jenkins organization, and they have already done some contributions.
This year we have the following projects:
Code Coverage API Plugin - create a new API Plugin to unify existing Code Coverage plugins and provide new features.
Student: Shenyu Zheng from Henan University, Kaifeng, China
Mentors: Steven Christou and Supun Wanniarachchi
Electronic Design Automation Plugins - create plugins for open-source Electronic Design Automation tools for synthesis, simulation and coverage analysis (iVerilog, covered, Yosys).
Student: Udara De Silva from University of Akron, Ohio, USA
Mentors: Martin d’Anjou and Oleg Nenashev
Jenkins Remoting over Message Bus/Queue - add support of a popular message queue/bus technology (RabbitMQ or Kafka) as a fault-tolerant communication layer in Jenkins.
Student: Pham Vu Tuan from Nanyang Technological University, Singapore
Mentors: Oleg Nenashev and Supun Wanniarachchi
Simple Pull-Request Job Plugin - add ability to define Jenkins jobs as YAML files stored in SCM, integrate it with existing plugin ecosystem.
Student: Abhishek Gautam from Visvesvaraya National Institute of Technology, Nagpur, India
Mentors: Jeff Knurek, Kristin Whetstone and Willy Aguirre
During next 4 weeks project teams will be reaching out to potential stakeholders in order to establish connections and to get comments regarding their project designs. If you are interested in the projects, please join discussions in the Developer mailing lists and project meetings once they get scheduled. Please also expect expect more detailed blogposts about the projects soon.
If you are interested to know more about GSoC in Jenkins, you can find information, timeline and communication channels here.

A good backpack will hold all your stuff. A great backpack will keep your stuff out of thieves’ hands.
This sort of slid under the radar in the middle of some bigger changes for the JEP-202 reference implementation, so I wanted to call it out now. Arguably this could deserve a retroactive JEP, though I would rather fold it into a JEP for JENKINS-49651 (see below).
As of Jenkins 2.118, or plugin parent POM 3.7, you can mark any Java member (class, method, constructor, field, or I suppose also interface, enum, or annotation) with API visibility (protected or public) with an annotation:
@Restricted(Beta.class)
The idea is to announce to potential users of the member that the API may still be in flux and only code prepared to keep up should be using it. For an example, 2.118 added a VirtualFile.toExternalURL() method that is being implemented in artifact-manager-s3 and (pending some PR merges) called in copyartifact and workflow-basic-steps. We do not necessarily want this to be called yet by unknown parties out there in the Jenkins ecosystem. To enforce that, any attempt to call or implement toExternalURL will produce a build failure, unless you add this property to your plugin POM, as these plugins have done:
<useBeta>true</useBeta>
Why? Because there is a chance the design is wrong and it might need to be changed—perhaps some upcoming bug fix would demand a boolean parameter be added, for example.
Under the conventional notion of Jenkins API deprecation and compatibility policy, once an API like this makes it into a release version, that is it—we might mark it @Deprecated but we need to maintain compatibility indefinitely, and find some way to migrate existing implementations / call sites.
With the @Beta annotation, that promise is not being made. If it needs a boolean parameter for some reason, that will be added and those three plugins updated to match; we are not going to bother retaining the original overload and somehow delegating to the new one. This simplification of the developer workflow is important to the use cases of Essentials (JEP-3xx), and I would expect the useBeta mark to become widespread among plugins included in Essentials. Such as the situation where one team needs to feel comfortable refactoring code under its aegis freely, and the refactored result should be deliverable as a unit to production via the Evergreen distribution system.
So that leaves two important questions:
First, is the annotation permanent, and if not, when should it be removed? I do not think there is any hard policy, but the intention is that it should be removed once the API is in more or less widespread use and has held up. For this example, if people start using S3 artifacts, and especially if someone successfully writes an implementation of artifact storage in Azure that uses the API, the concept will have been reasonably proven. At that point we want the API to be used wherever it would make sense, and if there is some very belated realization that the design is not quite right, we accept the burden of deprecating the original and migrating callers compatibly.
Second, it is fine and well to say that someone changing the signature of a beta toExternalURL is on the hook to update the three plugins using it, but what if a Jenkins admin (not running Essentials, for shame) upgrades to (say) Jenkins 2.125 with the new signature but declines to accept the updates to those plugins (say, workflow-basic-steps 2.9) which adapt to the change? It is not enough to say that it is their fault for holding back on the updates arbitrarily; the plugin manager offers you updates but does nothing to tell you when they are required, so suddenly throwing NoSuchMethodError is not a helpful response.
The solution needs to be ironed out, but my expectation is to use JENKINS-49651 for this. For example, workflow-basic-steps 2.8, using toExternalURL(), would have declared itself compatible with Jenkins-Version: 2.118, and thus implicitly anything newer. The developer doing the refactoring would also amend some 2.125 (and newer) core metadata to say that it conflicts with anything older than the 2.9 release of the plugin. The plugin manager would therefore block the 2.8 plugin from even being loaded on the 2.125 core; the admin would need to update before using it. In the case of an incompatible change made to a plugin API, rather than a core API, the UX is a little smoother since the plugin manager could just refuse to let you update one without the other.
If you’re a plugin or core developer who is interested in using the @Beta annotations, or have questions about our motiviations, please join the discussion on this mailing list thread.

There are, quite literally, hundreds of Android phones out there, so picking the best one can be a challenge.