Easy way to screw up leadership in your team

Let’s start with a hypothetical situation. You are one day working as always and one of your direct reports needs to talk. Let’s assume you drop everything and can jump into a call. Your teammate needs some help: he applied to a conference and needs some day off to attend. This should be an easy one, conferences are fun and people always learn in conferences (this blogpost can also apply to a medical procedure or something similar). There is a trick, your project and your client (could be internal/external) might be impacted by this decision. Maybe there is not a lot of runway to make the decision.

What do you do now? The easy way is: “Sorry, we can’t do it this time. There is an important deadline/deliverable and we need you to reach it”. This is the easiest way out, you don’t need to talk with the client and there is always a conference next year. Maybe next year planets will align?

There is another way. You say: “I got your back. This conference is important, is a great opportunity and I will fight whoever I need to fight for you to take the days”. How confident you sound and how much you can promise will depend on your position in the organization and how much influence do you have. Sometimes you might need to say: give me some time and I’ll try my best. But what you need to do after that is fight with all that you have for him. Think about creative ways to reach the goal, maybe some overtime before the conference or maybe the whole team can help. 

The result is important but it is not everything. What matters is how you communicate and that your teammate feels that you are on his side. This can make or break a relationship. In tech job markets are always hot, you will not be able to always compete in salary but these little details can make the difference. 

Also there is something way worse you can do. Please never do this. You can blame the teammate. It’s sad but I’ve seen it in the past. Sometimes managers say: “Why did you do this? If I bring this up it can bring a lot of trouble. This is a big headache“. Please don’t do it. At least try to get the days for your teammate. 

How would you handle this situation? Have you had different experiences? 

Standard

How do you find your weak ties?

Weak ties are very important in your life. You can find some science here. Weak ties are people that are in your network but you don’t talk to them very often. Your strong ties are very important but you’ll get very interesting and more novel information from weak ties than your strong ties.

In the past, I found my weak ties in the office or simply by going out to the supermarket, shopping mall, or restaurant. Today because of the time we are living in (lockdowns and a pandemic) it’s hard to find your weak ties. Being most of the time at home reduces the chances to find those people.

The other place you might be able to find them is online, but this is a complex topic. I feel that leaving the chances of finding a weak tie to an algorithm has not worked very well in the past. There is a chance you find someone while you scrolling, but I guess those algorithms might be optimized to show stuff from your strong ties (to increase engagement and the chance of showing you ads). This could be incorrect but I don’t think there is a lot of value in optimizing for weak ties (or is it?).

I’ve been thinking about how to find these people in a more disciplined and systematic way. There might be an app already or one could use a spreadsheet. I would like to have a way to log the people I interact with on a daily basis. After some time one could find strong and weak ties, and try to reach out to the weak ties. Sometimes just saying Hi to a weak tie is nice.

This method might not help to find all the weak ties. It could leave the very important ones but is a start. Which method are you using today? Happy to hear about it in the comments or on Twitter.

Standard

From Networking to Machine Learning in just 10 years

2020 marks the end of my first decade working. It has been a great ride. When it started I had a very different idea of what was going to happen. I have learned a lot. For some reason, I never optimized for money (the journey would have been very different), always tried to do what looked fun, and followed my gut. I would not recommend anyone following a similar path, just wanted to share it in case you find something interesting.

  • College: I studied Telematics because it did not want to be in front of a computer all the time. When we are young, we are naive and for some reason, I did not want to study software engineering. In college, I enjoyed a lot the Networking courses, Math and Physics. I took 4-5 programming courses and they influenced my working life a lot. 
  • First Job: I worked for 3 years at a local Internet Service Provider. I learned a lot about leadership, networking, and professional life in general. My first code to run in production was a bash script I hacked in 45 mins, fortunately, it worked. At some point, I felt Networking was more about using technology than creating it. I started dreaming about programming and starting a startup. 
  • Startups: I started a couple of startups after my first job. One was a SaaS product for tourism operators the other one was a tiny consulting firm in IoT. The main takeaway here was that I started programming full time. I had to learn a ton of stuff on the fly and was not very good at the beginning. I had some great teachers and was an avid user of StackOverflow. I thought the IoT startup was a product company, but we were actually in Consulting (I learned this like 5 years later). I had the chance to create some interesting products and watch them in production. I still feel happy that we did not break anything serious. I was young, reckless, and always said yes when people asked me if I could do something. There were a lot of long nights and fixing stuff on client datacenters. At the same time, I taught some classes at a local college (Mostly about networking and computer architecture). 
  • First Consulting Job: After the startups, I needed some money and wanted a more regular job. By some interesting coincidences, I landed at a software Consulting firm that was opening up operations in my country. It was a fun ride. We did fairly large development projects and I learned a lot about leading and managing software projects. We worked at the client office and this taught me a lot. I also learned a lot about the Consulting business, given that it was a small operation there was an opportunity for me to learn and help in different parts of the business. I enjoyed helping with invoicing and other administrative tasks. I also burned out, grabbed more work than what I was ready to take. I learned some key lessons about worked in general. 
  • Second Consulting Job: Before my second Consulting Job I did a quick pause to teach some more courses and learn other things (Executive Coaching). This pause helped me recover from burn out and prepare for my next adventure. I started taking some courses on ML (still remember when I finished Andrew’s Ng ML course). I got hired in my second Consulting Job to work on RPA projects. By pure luck, there was an opening in a new ML team. I jumped at the opportunity and kept studying. Had the chance to put a model in production and be with the team the whole process, from whiteboard to production. Also had the opportunity to work with some super smart Ph.D.’s and they were also very humble. I enjoyed working with the team a lot and learned a ton of lessons.

This is where I am today 10 years after starting to work at the ISP. I have enjoyed a lot this journey and am super excited about what is coming now. I am not sure if in 10 years I’ll be in the same place working on the same kind of problems. Hopefully, I’ll be able to come back to this piece and look back with a smile. 

If you have any questions/comments, feel free to reach me on Twitter.

Standard

Leading someone with more years of experience than my age

It all started at my first job. I’ve been around 6 months in the company and was happy getting used to the ropes of my current job. In those 6 months, I quickly pushed to prod (without even knowing it) and was learning very fast. I still remember when my boss called me to his office: We are going to do some changes and you’ll lead this new 3 person team. I thought to myself: Cool, I’ve learned some books about leadership, this could be fun.

The tricky part was the members of the team. They were a couple of technicians who had more than ten years on the job and more than twenty years in the company ( I was twenty years old). One of them was even a team lead recently. I did not understand the mechanics of that re-org at the time and still don’t have all the information about what was happening. I wish I had some cool story about why I was chosen to lead the team, really don’t know.

It was not a walk in the park, things started to get ugly before they could get better. As expected they pushed back a lot, they did not like the change, me, or anything that was happening. It’s funny, looking back I think my ego should have been hurt based on all the times they tried to avoid working with me and change things. I guess I was so confused with everything that I had no time for that.

When we started to work together, one thing was clear: they were not in the mood to take orders from a kid. I was not in the mood to give orders to them. This particular situation helped me a lot and shaped the way I work ten years later.

Given that it was almost impossible to give orders and avoid starting a world war. I started listening, that’s what I did the most in the first weeks. I listened and asked questions. Not like interrogations, more like curious questions. I think the key to the questions what that I wanted to learn from them. I was not a trick, I was very interested in what they were doing.

I still remember some kind of pranks they pulled on me. They would tell me they had run out of a particular material they needed. I had made a lot of calls and research to try to find it for them, then they magically found some in the warehouse or the car they used. I did not know these were pranks, I learned later when they had another team lead (I laughed that time).

With some retrospective, I think these pranks were not just to pick on the new guy. I think they were testing me. They wanted to see what I did, would I forget about the problem, slack off, blame it on them, or take ownership and be part of the team. Most of the time I came with solutions and discussed alternatives with the team. Some weeks later I think they started to trust me.

There was a point where I started to notice that I could help with some things. Not so much in executing, but there were some things they needed help with. Mostly operational excellence. They were relieved to have someone handle that. Every week we were becoming a better team.

Patience was a foundation for this team. At least from my side. I did not rush things out (I was not even sure how to do that) and never try to impose my ideas or way of working. I trusted a lot the team and things started to look better.

Not everything was perfect. We had disagreements and sometimes we preferred different approaches. This is normal in all teams, the good part is that we started to have a process. We had a daily meeting, talked about what we needed to do that day and we talked briefly at the end of the day. Every side learned to cede sometimes and when months have passed we were playing by ear.

Even though we were a productive team and technically very respected, what I always treasure from these experiences is human relationships. I still have a great time whenever I talk to these two people and we have a blast remembering the good old days. Based on what they say, they learn a thing or two from me, but I learned a ton from them and that experience.

I am writing an ebook about how you can choose a problem to solve and make money. Please let me know any feedback on Twitter.

Standard

A superpower every dev and data scientist should have

When you start your career in tech a big part is finding your superpower. At the beginning you might have to try different things to know what is yours. Some people are super happy working on databases, others like backend stuff, machine learning… you name it. A big part of your 2-3 first years is finding the intersection between what you like and what you are good at.

It’s normal that we look into tech-related superpowers. It’s what we do most of the day. But there is another superpower that can make you stand out. Does not matter if you are a developer on a big company, founding a startup or somewhere in between. I’ve found that this superpower can change your career and help you grow in some interesting directions.

The superpower is talking to business people or to other people in general. You might not like it but it can help a lot. You might need to communicate with them for different reasons: you might want to suggest a new feature, you think the team should spend more time on a specific topic or you might need to explain a delay in the development process ( I know this never happens :-)).

Understanding how much to talk, what messages to pass on, and how to structure them can be key in your professional development. And even more importantly, can make you enjoy your job way more. People will enjoy working with you, they’ll look to you for advice and you will enjoy working with them.

You don’t need to be a smooth talker. Just focus on your audience, what are they interested in? What do they need to understand? Don’t use big words. Don’t lecture them in Computer Science or Statistic complex concepts. Understand their metrics, what they are trying to achieve, and align your speech to those.

It’s never the same to say we need to implement SAML2 than to say: single sing on will make our customer life easier. Don’t say we need to re-vamp our whole backend to improve performance, say: we’ll lower our order processing on an average of X%. Those little tricks can help you get your features built and explain better what you are trying to achieve.

Standard

Before starting a Data Science project in the company you must…

You’ve probably heard about Data Science, Machine Learning, AI and all those cool buzzwords. You have heard that some of those technologies can help you use your data to predict the future: forecast your demand, avoid fraud and recommend products to customers. It’s great that you are thinking about starting one of these projects.

Most of the times teams start thinking about the how. They focus on the algorithm, on which kind of problem they are trying to solve. Is this a classification problem, clustering or regression? This might bring some undesirable results to the project. The team may optimize the how but forget about the fuel that makes it all run: data.

Before starting you should work on a Data Quality report. It does not needs to be very complex but it needs to answer the following questions:

  • Can I trust this data?: Do you know where this data comes from? How was it collected? How much human intervention has it gone through? Is it external or from one of my own systems?
  • Do someone on the team understand this data?: Are we sure we know the unit of measurements this data should have? Does the frequency makes sense for the kind of problem we are trying to solve? Can somebody tell if an specific data point is off the charts? (Imagine having a negative number on a sales table)
  • Is the data good? There are lots of frameworks to evaluate this, but you can start with simple questions. Do I have a lot of empty rows? Do I have a lot of repeated data? Are the data types correct for all my dataset.

Once you are able to answer this three group of questions you will be in a better position to start your project. During this excercise you might find that you wont be able to use some of the data or asking around you might found about new data the you haven’t event thought about.

Standard

Detecting soccer teams using unsupervised learning and tensorflow object detection (images and videos)

In the past we have used Tensorflow Object Detection to detect sharks, social distancing and squirrels. Detecting objects is fun and we can build on top of that. Our main task will be to detect the two teams on a soccer field. We will use Tensorflow Object Detection to detect the people and then we’ll use unsupervised learning to cluster the people objects based on their shirt color. We’ll use k-means to cluster the people objects.

We’ll start with the regular Tensorflow Object Detection sample. After that we’ll follow some steps to build our little project.

This will be our end result:

First thing we’ll need to do is modify the method: visualize_boxes_and_labels_on_image_array . This will allow us use a different bounding box color for each team. Although we need to copy-paste the whole method, the change is pretty small:

        '''
        if agnostic_mode:
          box_to_color_map[box] = 'DarkOrange'
        elif track_ids is not None:
          prime_multipler = _get_multiplier_for_color_randomness()
          box_to_color_map[box] = STANDARD_COLORS[
              (prime_multipler * track_ids[i]) % len(STANDARD_COLORS)]
        else:
          box_to_color_map[box] = STANDARD_COLORS[
              classes[i] % len(STANDARD_COLORS)]
        '''
        box_to_color_map[box] = STANDARD_COLORS[team[i]]
        

We commented a lot of stuff and assigned the color based on a team array that contains different numbers for each team.

Then we’ll have our main method which will let us detect the teams. At a high level this method performs the following steps:

  • Performs object detection and filters people
  • Processes the coordinates to feed them into the k-means
  • Use k-means to find clusters
  • Displays the images with the teams detected
def detect_team(model, frame,df):
  # the array based representation of the image will be used later in order to prepare the
  # result image with boxes and labels on it.
  
  person_class = 1
  original_image = frame
  
  image_np = frame
  # Actual detection.

  output_dict = run_inference_for_single_image(model, image_np)

  boolPersons = output_dict['detection_classes'] == person_class
  output_dict['detection_scores'] = output_dict['detection_scores'][boolPersons]
  output_dict['detection_classes'] = output_dict['detection_classes'][boolPersons]
  output_dict['detection_boxes'] = output_dict['detection_boxes'][boolPersons]

  r_points = []
  b_points = []
  g_points = []    


  for i in output_dict['detection_boxes']:
    new_box = denormalize_coordinates(i,original_image.shape[1],original_image.shape[0])
    im2 = original_image[int(new_box[0]):int(new_box[2]),int(new_box[1]):int(new_box[3]),:]
    r_points.append(im2[:,:,0].mean())
    b_points.append(im2[:,:,1].mean())
    g_points.append(im2[:,:,2].mean())

    new_row = {'R':im2[:,:,0].mean(), 'G':im2[:,:,1].mean(), 'B':im2[:,:,2].mean()}
    df = df.append(new_row, ignore_index=True)

  #print(df.shape)
  if len(output_dict['detection_boxes']) > 1:
    kmeans = KMeans(n_clusters = 2, init = 'k-means++', max_iter=1000, n_init = 100, random_state=0)
    y_kmeans = kmeans.fit_predict(df)
  
    visualize_boxes_and_labels_on_image_array(
      image_np,
      output_dict['detection_boxes'],
      output_dict['detection_classes'],
      output_dict['detection_scores'],
      category_index,
      instance_masks=output_dict.get('detection_masks_reframed', None),
      use_normalized_coordinates=True,
      line_thickness=8,
      team = y_kmeans)
    '''
    fig = plt.figure()
    ax = fig.add_subplot(111, projection='3d')
    ax.scatter(r_points, b_points, g_points, c=y_kmeans)
    plt.show()
    '''
  return image_np

Another interesting part is how we apply k-means. Given that images in numpy are represented with a tridimensional vector (red, green ,blue) we average each layer and get 3 numbers per people object. We feed those 3 dimensions into the k-means and get the clusters.

You can also display the k-means visualization by uncommenting these lines:

    fig = plt.figure()
    ax = fig.add_subplot(111, projection='3d')
    ax.scatter(r_points, b_points, g_points, c=y_kmeans)
    plt.show()

I also added a code snippet that you can use to read a video and generate another video with the detected teams:

from google.colab.patches import cv2_imshow
import cv2
FILE_OUTPUT = "test.avi"

PATH_TO_TEST_IMAGES_DIR = pathlib.Path('models/research/object_detection/test_images/soccer.avi')

vcap = cv2.VideoCapture('models/research/object_detection/test_images/soccer.avi')
frame_width = int(vcap.get(3))
frame_height = int(vcap.get(4))

out = cv2.VideoWriter(FILE_OUTPUT, cv2.VideoWriter_fourcc('M', 'J', 'P', 'G'),
                     24, (frame_width, frame_height))
ret, frame = vcap.read()


i = 0
while(i<1):
    ret, frame = vcap.read()
    im = detect_team(detection_model, frame,df)
    #cv2_imshow(im)
    out.write(im)
    i = i+1

vcap.release()
out.release()

Take a look at the video:

You can find the code on this repository.

Standard

Object Detection. A shortcut when thinking about labeling images

Detecting objects on an image can be accomplished by using a deep learning model. There a lot of pre-trained models on the Internet that you can use. Sometimes you might want to train your model to detect a specific object (sharks, squirrels, a mask on a person’s face…). There are multiple tutorials about how to train these models on custom datasets. Don’t.

Before even thinking about creating your own dataset. Downloading 100s of images and labeling them using labelimg can take a lot of time. And in some cases, it might be unnecessary. Test some out of the box models before going into the long route.  This step won’t take long and can save you a ton of time.

I am going to use a couple of examples. I want to detect certain animals in pictures. Sharks and squirrels. Let’s say we have a research purpose to do this. Before using the long route let’s try the other approach.

We are going to use the official Google Collab notebook to test the different models. The notebook is pretty straight forward if you run all the cells you are going to use a common model trained on a dataset called Coco. The full name of the model: ssd_mobilenet_v1_coco_2017_11_17.  This is a fast model but looks like it won’t work for our purposes. I uploaded an image to /content/models/research/object_detection/test_images and this is the result we got:

squirrel mscoco 3.png

Not the results we were expecting. It has a low confidence and not a very good prediction. If we assume that’s a squirrel.

Before changing some stuff on the code, you can find some pre-trained ready to use models on the Tensorflow detection model zoo. There are models trained on different datasets and with different performances.

Let’s change a couple of lines of code and test again. First we are going to use a model from the Inaturalist dataset:

  • On the section “Loading Label Map” we are going to use the following code: PATH_TO_LABELS = ‘models/research/object_detection/data/fgvc_2854_classes_label_map.pbtxt’  The label map helps us interpret the output of the new model. It ties the category number to a name.
  • On the section “Detection” we are going to use the following code: model_name = ‘faster_rcnn_resnet101_fgvc_2018_07_19’ This will tell the code which model to download.

With the new model this are the results:

download.png

Higher confidence and a weird latin name (scriurus carolinensis). If we use wikipedia we find that the other known name is Eastern gray squirrel. Not bad. The same if we test a shark image:

whale shark naturalist.png

Using Wikipedia we can find that it’s a whale shark.

Just for the same of experimenting I used a model trained on the OpenImage dataset. I used the following label map and model:

  • model_name = ‘faster_rcnn_inception_resnet_v2_atrous_oid_2018_01_28’
  • PATH_TO_LABELS=
    ‘models/research/object_detection/data/oid_bbox_trainable_label_map.pbtxt’
shark open image.png

Here we got a higher confidence but not as accurate detection. Depending on your use case one model or the other could be better.

In some cases you’ll still need a custom dataset and going the long route. But checking this avenue won’t hurt and might save you a lot of precious time.

Standard

How to survive multiple interview processes

As Paul Krugman says: These are strange times. We are living under unusual circumstances and there are a ton of negative stuff to think about. A lot of people have been laid off of their jobs and are looking for a new gig.

For software developers we have an interesting market, some companies are reducing the headcount and some others are hiring. For junior or mid software developers there is an additional challenge they might fit in different positions based on the programming languages they know or like. Which one should I study? Should I polish the ones I already know? Should I learn a new one? As always the answer is: it depends. I’ll elaborate on the following paragraphs.

I’ve seen a lot of similarities between finding a new job and selling enterprise software (please don’t run yet, bare with me). In sales we have a pipeline were we manage opportunities and try to maximize our sales numbers. Here you only need 1 win, only need to be right once. I’ll explain in simple terms what’s a pipeline and how you can use it to survive this tough situation.

The pipeline has 5 stages. We have some interesting assumptions and facts:

  • Not all opportunities have to go through the 5 stages.
  • As we advance in the pipeline the probability of winning should increase.
  • Opportunities have different velocities to the pipeline, some can change really quickly and others can be very slow.

The 5 stages are:

  1. Identify: Here opportunities are like gossiping. You heard a company is hiring, you read a post on LinkedIn. At this point we know there might be an opportunity but you don’t even know if you have what it takes.
  2. Qualify: To be able to pass to this stage you must assess if the opportunity is a fit for you. Review the seniority level they are looking for, years of experience, programming languages and industry.
  3. Pursue: Once you send your application (CV) or you asked a friend to refer you, then we are on pursue. By this point, it might be a good idea to polish or learn new languages.
  4. Closing: Now we are in the interview phase. This can vary a lot from company to company: you can have several phone interviews, in the past, you might have an on-site and you could be interviewed by your future boss, to name a few.
  5. Won: You accepted the offer! Congrats! Hopefully, you’ll be on this stage pretty soon.

After that long explanation, I would say you should study new languages or polish the ones you know based on your pipeline. Probably doing an Elixir course based on an opportunity in the Identify stage could not be a great idea (it could if you really want to work on Elixir). You could get serious about a new language if you are on Pursue or Closing.

This advice might not apply to everyone and every situation. It’s my 5 cents to try to help in this difficult time we are living.

Good luck finding your next gig!

Prueba001-05.jpg

30 things I learned at my first job (Daniel Rojas)

 

Standard

How to create a practical Open AI custom environment for the rest of us (sourcing problem)

Why create a custom Open AI?

Today we are using classical Machine Learning and Neural Networks to solve all kinds of problems. Reinforcement Learning is being used to solve games and some industrial applications, but I think this will change pretty soon. Instead of tagging images, we will be creating business environments for our agents to learn and perform in real life. I’ve read some tutorials on how to create an environment but the framing is the difficult part.

The sourcing problem

I wanted to build an environment that had to do something with business. All the companies have sourcing departments. People who buy stuff, either to resell it (as e-commerce) or to operate the business (you need pens, cars, Dunder Mifflin paper…).

When you buy the stuff you have to make decisions. Imagine that you have 5 suppliers, each one has a different price and different reliability. There are very cheap suppliers but not very reliable and there is a premium that always delivers on time. You can find the example on the following table:

Screen Shot 2020-04-12 at 11.07.22 AM.png

In this case, paying a low price carries a high risk. Every supplier sells you 5 articles at a time and you only have $1000 to buy as many articles as you can. The best-case scenario would be that you are super lucky and buy always

Framing the sourcing problem as a Reinforcement Learning problem

Before start writing the code, there are some things we need to define. In my case this takes me more time than writing the actual code:

  • Action space: Which actions can your agent take? Here we will have 5 actions, buy from 1 of the 5 suppliers.
  • Observation space: This is what our agent will see. In our case, we will let him know how many articles he currently has, how close he is to the goal (50 articles).
  • Reward: You need to give a prize to your agent when he does good. Here we will use the following function: (current articles/max_articles:50)^0.9

Screen Shot 2020-04-12 at 11.12.22 AM.png

For me, the trickiest points here are the observation space and the reward function. In the observation space, I tried to read a lot of code from existing environments in Open Ai documentation. In this table, you can find environments and their types so you can look faster. Choosing the reward function is an art, I watched a great youtube video that gave me the idea of choosing this function.

Simple steps to create a custom Open AI Environment

Once you framed the problem in an RL way it starts going downhill.

  • You need to create the file structure for your environment. There is great documentation on Open AI. Be careful with the names and make sure you replace everything, this can get cumbersome fast.
  • After you’ve done this you need to modify some specific methods:
    • __init__: where you initialize all the variables.
    • step: whenever your agent takes action, it will call this method. The environment will return a state so the agent can take its next decision.
    • reset: Once you’ve reached a terminal state (you are out of money or bought all the articles) you need to reset everything that way the agent can try again.

Creating an agent

To test the environment I used q-learning. Explaining this is outside of the scope of this blogpost. I used the code from this great tutorial, I recommend watching his videos in case you want to learn more about Reinforcement learning. You can find the code here.

Results

I ran the agent with different training episodes and plotted how he performed.

With 1 training episode (almost random):

The lowest action is riskier but the cheapest. In this example, it managed to buy 30 articles.

Using 600 training episodes it managed to solve the problem. It was able to buy 50 articles and use an average approach.

Code

You can find the code on this repository.

I’ll be happy to hear about experiments, ideas or questions on twitter

Prueba001-05.jpg

30 things I learned at my first job (Daniel Rojas)

 

 

 

Standard