Magento 2 Attribute Groups REST API

A common problem that developers run into when developing against Magento 2 APIs is that the documentation is not very good. A lot can be done with an OpenApi (formerly Swagger) spec and the Swagger UI tool is great for testing, but when forces in the world are shifting more and more towards mobile development, the strength and use of APIs becomes much more important.  Along with that, documentation is key, but is often times done as an after thought or auto generated from code. Regardless of that, this blog post is about one particular problem that I have with a particular API call.

Magento has an Admin REST API which accepts the standard searchCriteria parameters with the route:

GET /V1/products/attribute-sets/groups/list

Based on the definition in the spec and the limited documentation, one would expect that this would allow for searching across all attribute groups, or to simply get a list of all attribute groups that are defined within the system.  Using the search parameters as one would expect will result in an error with the response message:

{
  "message":"%fieldName is a required field.",
  "parameters":{
    "fieldName":"attribute_set_id"
  },
  "trace":"STACKTRACE INFORMATION"
}

This makes no sense because attribute_set_id is not one of the parameters and is a source of confusion across the interwebs.

As it ends up, this call actually only allows for listing all attributes that belong to a particular attribute set.  The “search parameters” are not really search parameters as one expects and only work if set as such:

searchCriteria[filterGroups][0][filters][0][field]=attribute_set_id
searchCriteria[filterGroups][0][filters][0][value]={{SOME_ATTRIBUTE_SET_ID}}

Any other parameters seem to be ignored and are meaningless.

This combination of path and parameters is very confusing and does not follow a standard REST collection/resource design pattern.  There is another route that is a PUT /V1/products/attribute-sets/{attributeSetId}/groups which has a more logical path that could be used for this GET call, but this is unfortunately not how the world is, but hey, the future is coming.  🙂

One simple suggestion for the Magento team: why not have wiki documentation for the APIs as well as just the spec and some simplistic examples?

How to install an enhanced LAMP stack on Ubuntu 16.04

Script

This post describes the steps in detail, but please see the Git Hub project that performs these steps in a script here:

https://github.com/deviantlycan/ubuntulampscript

Assumptions

  • System is running Ubuntu 16.04
  • Chrome installed
  • This is being installed on a local computer.  In the case that it is remote with command line access only, replace the system’s IP address where instructions indicate “localhost”.  Additionally, when installing remote, the chrome commands will not work and you will not be able to use MySQL Workbench

Prep the system

  • sudo apt-get update
    • Updates packages
    • Enter password when prompted
  • sudo apt-get upgrade
    • Updgrades packages to the latest versions
  • sudo apt-get autoremove
    • Removes old packages that are no longer in use and cleans up disk space

Install Chrome

  • wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
  • sudo dpkg -i –force-depends ~/google-chrome-stable_current_amd64.deb
  • sudo apt-get -f install -y
  • sudo dpkg -i –force-depends ~/google-chrome-stable_current_amd64.deb

Install Apache

  • sudo apt-get install apache2
    •  Installs Apache 2
  • sudo systemctl enable apache2
    • Enables Apache 2 on the system
  • sudo systemctl start apache2
    • Starts the Apache 2 server
  • systemctl status apache2
    • Shows the status of the running instance of  Apache
    • Press ‘q’ to exit
  • /opt/google/chrome/google-chrome http://localhost/
    • If chrome is installed, this will start a browser and load the default Apache page to validate that it is serving content as expected.

Install MySQL

  • sudo apt-get install mysql-server mysql-client
    • Installs MySQL command line client and server
    • Set the root password when prompted
  • sudo systemctl status mysql
    • Validate that mySql is running.
    • Press ‘q’ to exit

Install PHP 7 and tools

  • sudo apt-get install php7.0-mysql php7.0-curl php7.0-json php7.0-cgi php7.0 libapache2-mod-php7.0 php-mbstring php7.0-mbstring php-gettext php-soap php-imap php-xdebug php-ldap
    • Installs PHP 7.0 and other useful php extensions
  • php -v
    • Validates that PHP is successfully installed
  • echo “<?php phpinfo(); ?>” | sudo tee -a /var/www/html/testphp.php
    • Creates a php test file that will output php configuration info.
  • /opt/google/chrome/google-chrome http://localhost/testphp.php
    • Opens the php test page in chrome
  • sudo curl -s https://getcomposer.org/installer | php
    • Download and install Composer
  • sudo mv composer.phar /usr/local/bin/composer
    • Move Composer to a global location
  • composer
    • Run composer to make sure it is installed correctly.
  • wget https://phar.phpunit.de/phpunit-6.3.phar
    • download PHP Unit (for PHP unit tests)
  • chmod +x phpunit-6.3.phar
    • Make it executable
  • sudo mv phpunit-6.3.phar /usr/local/bin/phpunit
    • Move it to a global location
  • phpunit –version
    • run php unit to make sure it is working

MySQL Tools

If you are working locally, you can install the desktop tool MySQL workbench for a GUI interface into the MySQL instance.

  • sudo apt-get install mysql-workbench

If you want to have easy to use remote access to administer the database, you can install phpMyAdmin

  • sudo apt-get install phpmyadmin
    • Select apache2
    • Select Yes when prompted for “Configure database for phpmyadmin with dbconfig-common?”
    • Enter the administrator password for MySQL when prompted.
    • Confirm the password when prompted.
  • echo “Include /etc/phpmyadmin/apache.conf” | sudo tee -a /etc/apache2/apache2.conf
    • Enables phpMyAdmin
  • sudo systemctl restart apache2
    • Restart Apache
  • /opt/google/chrome/google-chrome http://localhost/phpmyadmin/
    • Loads the phpMyAdmin login screen in Chrome
    • Log in with the username “root” and the root password entered when setting up MySQL

Install Memcache

While not officially part of a basic LAMP stack, Memcache is super useful and highly recommended.

  • sudo apt-get install memcached php-memcache php-memcached
    • Install memcache server and the php memcache extension
    • configuration file is located here /etc/memcached.conf
    • Optionally increase the amount of memory that memcache uses.  Default is 64 mb, which may be fine for a test server, but not for a production server.
  • sudo printf “extension=php_memcache\nextension=php_memcached” | sudo tee -a /etc/php/7.0/apache2/php.ini
    • Includes Memcache in PHP
  • sudo systemctl restart memcached.service
    • Restart the memcache service
  • sudo systemctl restart apache2
    • Restart Apache to load memcache modules
  • /opt/google/chrome/google-chrome http://localhost/testphp.php
    • validate that memcache is displayed in the PHP info

Optional Apache Configuration

Optionally, you can tune some basic Apache PHP settings.

  • sudo gedit /etc/php/7.0/apache2/php.ini &
    • Opens the php.ini file for Apache in a gui editor
    • Some useful changes
      • date.timezone = America/Chicago
      • memory_limit = 512M
        • Gives php a half gig of memory to work with.  Depending on the application, this may need to be increased
      • file_uploads = On
        • This is usually already on.
      • upload_max_filesize = 20M
        • Sets the max upload file size to 20 mb
      • post_max_size = 20M
        • Sets the max post body size to 20 mb (for uploading large files)
      • max_execution_time = 300
        • Increases the max execution time for long running threads.
        • As a side note, if you need this, and you are a programmer, you really should consider spawning off a background thread to process whatever the heck you are doing.
    • Save your changes and close the file.
  • sudo systemctl restart apache2
    • Restart Apache to allow the new settings to take effect

Optional MySql configuration

  • sudo gedit /etc/mysql/mysql.conf.d/mysqld.cnf
    • Opens gedit with the mysql configuration file
      • character-set-server=utf8
      • collation-server=utf8_unicode_ci
      • max_sp_recursion_depth=100
      • max_allowed_packet = 20M
      • thread_stack = 512K
      • optimizer-search-depth = 0
  • printf “[mysqld]\nsql_mode=IGNORE_SPACE,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION” | sudo tee -a /etc/mysql/conf.d/disable_strict_mode.cnf
    • Disables strict mode by creating a new configuration file.
  • sudo systemctl restart mysql
    • Restart the MySql server

Install Selenium

Selenium is not a standard part of a LAMP stack either, but it is a great tool for testing applications built in a LAMP stack.  I highly recommend installing it also.

  • sudo apt-get install default-jre
    • Installs Java run time environment
  • sudo apt-get install default-jdk
    • Installs the Java Developers Kit
  • wget https://chromedriver.storage.googleapis.com/2.32/chromedriver_linux64.zip
    • Download the chrome driver for Selenium
  • unzip chromedriver_linux64.zip
    • unzip the package
  • rm chromedriver_linux64.zip
    • remove the archive
  • sudo mv -f ~/chromedriver /usr/local/bin/chromedriver
    • move to a global location
  • sudo chown root:root /usr/local/bin/chromedriver
    • make sure this is owned by the root user for security
  • sudo chmod 0755 /usr/local/bin/chromedriver
    • make the chrome driver executable
  • wget http://selenium-release.storage.googleapis.com/3.5/selenium-server-standalone-3.5.3.jar
    • Download Selenium
  • sudo mv selenium-server-standalone-3.5.3.jar /usr/local/bin/selenium-server-standalone.jar
    • Move Selenium to a global location
  • sudo chown root:root /usr/local/bin/selenium-server-standalone.jar
    • Make sure this is owned by the root user for security
  • sudo chmod 0755 /usr/local/bin/selenium-server-standalone.jar
    • Make it executable
  • sudo apt-get install xvfb
    • install xvfb for easily running Selenium
  • xvfb-run java -Dwebdriver.chrome.driver=/usr/local/bin/chromedriver -jar /usr/local/bin/selenium-server-standalone.jar
    • Runs Selenium with the chrome driver
    • ctrl+c to quit
    • This can also be run in debug mode by adding the -debug flag to the end of the command

How to install Docker on Ubuntu

Good Lord, looking for simple and straightforward instructions for installing and configuring Docker on Ubuntu had me googling all over.  Here are some straight forward instructions that you can simply copy and paste.

Open a command prompt and run these commands

sudo apt install docker.io
sudo apt-get install openssl ssl-cert
sudo mkdir /var/docker
cd /var/docker
sudo openssl req -newkey rsa:2048 -new -nodes -x509 -days 3650 -keyout key.pem -out cert.pem
sudo mkdir /etc/systemd/system/docker.service.d
sudo groupadd docker 
sudo gpasswd -a ${USER} docker 
sudo gedit /etc/systemd/system/docker.service.d/docker.conf &

Add the following to the file opened in gedit:

[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H fd:// -D --tls=true --tlscert=/var/docker/cert.pem --tlskey=/var/docker/key.pem -H tcp://127.0.0.1:2376

In your command prompt, continue with running these commands.

sudo systemctl daemon-reload

sudo systemctl restart docker

systemctl status docker.service

Docker should be running.  You should see output from the last status call that looks like this:

your-command-prompt:/var/docker$ systemctl status docker.service
● docker.service - Docker Application Container Engine
 Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
 Drop-In: /etc/systemd/system/docker.service.d
 └─docker.conf
 Active: active (running) since Wed 2017-03-01 11:33:15 CST; 8s ago
 Docs: https://docs.docker.com
 Main PID: 16141 (dockerd)
 Tasks: 17
 Memory: 15.3M
 CPU: 294ms
 CGroup: /system.slice/docker.service
 ├─16141 /usr/bin/dockerd -H fd:// -D --tls=true --tlscert=/var/docker/cert.pem --tlskey=/var/docker/key.pem -H tcp://127.0.0.1:2376
 └─16148 containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --shim containerd-shim --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --r

When running a docker pull command you may get an error like:

Warning: failed to get default registry endpoint from daemon (Cannot connect to the Docker daemon. Is the docker daemon running on this host?). Using system default: https://index.docker.io/v1/
Cannot connect to the Docker daemon. Is the docker daemon running on this host?

Be sure to run the docker pull with sudo like

sudo docker pull whatever/project:1.1.1-11

 

Very long string

Sometimes you need to test input with a long string and it is useful to easily see the index of each position so that you can see how much is read or tuncated or whatever.  Here is a handy string that I use.  It is divided up into 10 character blocks with no special characters or spacing (to see how wrapping will work or not work in some cases).  Each block starts with an “L” and ends with an “I” with “x”s and a number indicating position between.  This particular one is 1100 characters long, but it should be easy to extend or modify as needed.  Copy and paste is hella easier than making this by hand or using random characters that don’t indicate position easily.


Lxxxxxx10ILxxxxxx20ILxxxxxx30ILxxxxxx40ILxxxxxx50ILxxxxxx60ILxxxxxx70ILxxxxxx80ILxxxxxx90ILxxxxx100ILxxxxx110ILxxxxx120ILxxxxx130ILxxxxx140ILxxxxx150ILxxxxx160ILxxxxx170ILxxxxx180ILxxxxx190ILxxxxx200ILxxxxx210ILxxxxx220ILxxxxx230ILxxxxx240ILxxxxx250ILxxxxx260ILxxxxx270ILxxxxx280ILxxxxx290ILxxxxx300ILxxxxx310ILxxxxx320ILxxxxx330ILxxxxx340ILxxxxx350ILxxxxx360ILxxxxx370ILxxxxx380ILxxxxx390ILxxxxx400ILxxxxx410ILxxxxx420ILxxxxx430ILxxxxx440ILxxxxx450ILxxxxx460ILxxxxx470ILxxxxx480ILxxxxx490ILxxxxx500ILxxxxx510ILxxxxx520ILxxxxx530ILxxxxx540ILxxxxx550ILxxxxx560ILxxxxx570ILxxxxx580ILxxxxx590ILxxxxx600ILxxxxx610ILxxxxx620ILxxxxx630ILxxxxx640ILxxxxx650ILxxxxx660ILxxxxx670ILxxxxx680ILxxxxx690ILxxxxx700ILxxxxx710ILxxxxx720ILxxxxx730ILxxxxx740ILxxxxx750ILxxxxx760ILxxxxx770ILxxxxx780ILxxxxx790ILxxxxx800ILxxxxx810ILxxxxx820ILxxxxx830ILxxxxx840ILxxxxx850ILxxxxx860ILxxxxx870ILxxxxx880ILxxxxx890ILxxxxx900ILxxxxx910ILxxxxx920ILxxxxx930ILxxxxx940ILxxxxx950ILxxxxx960ILxxxxx970ILxxxxx980ILxxxxx990ILxxxx1000ILxxxx1010ILxxxx1020ILxxxx1030ILxxxx1040ILxxxx1050ILxxxx1060ILxxxx1070ILxxxx1080ILxxxx1090ILxxxx1100I

Derpy pathing

So, today I learned from a typo, that the GAE dev app server treats pathing defined in the @ApiMethod annotation differently than the cloud server.  In the dev app server, a path like either

path = “/endpoint”

path = “endpoint”

will resolve to

/_ah/api/whatevername/v1/endpoint

Where as the cloud GAE will resolve it as either

/_ah/api/endpoint

or

/_ah/api/whatevername/v1/endpoint

respectively.

This had me scratching my head for a while, but it is an interesting thing to remember and it may have some useful applications.

How to set up a proxy server on Google Compute (GCE)

There are times when a proxy server is necessary.  This article will not go into what a proxy server is or why you might need one, but this will give you step by step instructions on how to set up the squid proxy server on a Google Compute Engine (GCE) server.  This document assumes that you have a Google Cloud account and that you know how to set up a GCE server with Ubuntu.  Lines in a Fixed Width Font are commands that should be entered into the command shell.

This can be run on a GCE that has other things on it, but for personal use, a simple GCE f1-micro (1 vCPU, 0.6 GB memory) should be more than enough power.  Start your GCE and SSH into the machine to get a command line then do these simple steps.

  • Install squid
    • sudo apt install squid3
  • Install apache utils
    • sudo apt-get install apache2-utils
    • This Installs the htpasswd utility that allows for creating password files.
  • Copy the default configuration file to back it up.
    • sudo cp /etc/squid/squid.conf /etc/squid/squid.conf.original
  • Delete the old configuration file
    • sudo rm /etc/squid/squid.conf
    • This configuration file is ridiculously huge and has way more than any simple case would need.  We will just back it up and make a new one with only the information that we need.
  • Make a new configuration files
    • sudo vi /etc/squid/squid.conf
  • Enter this in the config file
    • auth_param basic program /usr/lib/squid/basic_ncsa_auth /etc/squid/passwords
      auth_param basic realm proxy
      acl authenticated proxy_auth REQUIRED
      http_access allow authenticated

      # Choose the port you want. Default is 3128, but we set it to 8888
      http_port 8888

    • The first bit (4 lines) sets squid to require passwords
    • The last bit configures squid to use port 8888.  You can change this to anything you want.
    • Save the file in vi with [esc]:wq
  • Set up the username and password
    • sudo htpasswd -c /etc/squid/passwords [username]
    • Replace [username] with any username you want to use
    • You will be asked to enter and reenter a password for the username that you entered.  make sure they match.
  • Restart the squid service
    • sudo systemctl restart squid.service

You now have a proxy server.  🙂  To use it, you will need to find the external IP address for this GCE and use that along with the port that you set (8888 in this example) and the username and password that you set.

To make life easier, you may want to set an alias for starting squid.  To do that, do the following:

  • Open your bashrc file
    • vi ~/.bashrc
  • Scroll to the bottom of the file and add this line:
    • alias startsquid='sudo systemctl restart squid.service'
  • Save the file
    • [esc]:wq
  • reload your bashrc file so that you can use the new alias
    • . ~/.bashrc
    • that’s (period)(space)~/.bashrc
  • now you can restart it with the alias.
    • startsquid